Editing FreeBSD Reference (section)

== ipfw on jail machines ==

The jail machines also have ipfw loaded, however all of the jail machines have ipfw loaded to default-accept.  This is because ipfw on the jail machines is not used for firewalling - it is used for traffic counting. Note- ipfw will not run inside a jail- a jail vps customer cannot have its own self-managed ipfw rules. You see, in addition to deny and allow rules, you can also do things like:

 ipfw add 00001 count ip from 10.10.10.10 to any
 ipfw add 00002 count ip from any to 10.10.10.10

which counts traffic bound from that IP, and bound to it.  Where does it count it ?  Simply in the ipfw rule itself.  Here is a sample from one of the jail systems:

<pre># ipfw show
00201        10          815 count ip from 198.78.70.176 to any
00202        10          802 count ip from any to 198.78.70.176
01631         1           62 count ip from 69.55.238.225 to any
01632         1          481 count ip from any to 69.55.238.225
01801        72        70154 count ip from 69.55.238.214 to any
01802        82         9047 count ip from any to 69.55.238.214
01811         3          245 count ip from 69.55.238.215 to any
01812         2          167 count ip from any to 69.55.238.215
01821         5          656 count ip from 198.78.66.216 to any
01822         4          377 count ip from any to 198.78.66.216
01841         0            0 count ip from 69.55.238.218 to any
01842         0            0 count ip from any to 69.55.238.218
01851         0            0 count ip from 198.78.66.219 to any
01852         0            0 count ip from any to 198.78.66.219
01861         3          218 count ip from 198.78.66.220 to any
01862         3          263 count ip from any to 198.78.66.220
01921         0            0 count ip from 69.55.238.224 to any
01922         0            0 count ip from any to 69.55.238.224
02241         0            0 count ip from 69.55.238.118 to any
02242         0            0 count ip from any to 69.55.238.118
02261         0            0 count ip from 69.55.238.223 to any
02262         0            0 count ip from any to 69.55.238.223
02271         0            0 count ip from 69.55.237.9 to any
02272         0            0 count ip from any to 69.55.237.9
02291        46        37023 count ip from 69.55.237.8 to any
02292        50        10939 count ip from any to 69.55.237.8
02311        20         1974 count ip from 69.55.237.7 to any
02312        22         1540 count ip from any to 69.55.237.7
03351         0            0 count ip from 69.55.237.163 to any
03352         0            0 count ip from any to 69.55.237.163
65535 102592563 113861945636 allow ip from any to any</pre>

Note two things - first, the packet counts and byte counts are very low, since this was taken shortly after system boot.  Second, notice that the last line is "allow ip from any to any".  That last line is the only line that affects actual traffic in any way - the others are just count rules.

Also, note that the rules are done by customer number - the customer number plus either a 1 or a 2 at the end - since every customer needs two total rules to count both inbound and outbound traffic. For instance:

 01811         3          245 count ip from 69.55.238.215 to any
 01812         2          167 count ip from any to 69.55.238.215

those are the two rules for customer 181 (col00181).
 
Remember how the `jailmake` utility asks for an "ipfw#" ?  The three or four digit representation of the customer number is what it is asking for. As you can see, the last section of jailmake contains these lines:

 /sbin/ipfw add `echo 0"$7"1` count ip from any to $ip
 /sbin/ipfw add `echo 0"$7"2` count ip from $ip to any

since the IP and the ipfw# are specified on the jailmake command line, this is very easy to do.  Note again that the rule number is prepended with a zero - which does not hurt anything if it is an extra zero in the case of a four digit customer number.
 
The jails do not add each individual ipfw line at boot time like the firewall does.  They are added by the postboot script which should only be run once to avoid duplicate ipfw entries.

The only odd thing about ipfw on the jails is that it is loaded as a module on jails 1-10, and loaded in the kernel in all jails 11 and beyond.

This is because doing traffic counting on the jails was not thought of until after we had loaded jail10.  So, rather than schedule a maintenance and reboot all 10 jail systems, we simply built a default-allow module, placed it in the / directory, and loaded that at boot time.  Therefore, on all jails 1-10, you will not only see something like this in the / directory:

 ipfw.4.7.accept.ko

but you will also see a line like this in /usr/local/etc/rc.d/boot.sh:

 /sbin/kldload /ipfw.4.7.accept.ko

(the module is named for the version of freebsd it was built for)

Jails 11-15 (and any future ones) have these lines:

 options IPFIREWALL
 options IPFIREWALL_VERBOSE
 options IPFIREWALL_VERBOSE_LIMIT=100
 options IPFIREWALL_DEFAULT_TO_ACCEPT

in the kernel configuration file - note the default-accept line.

The traffic counting on the freebsd machines is not just for our benefit -to see by running `ipfw show` ... there is also a cron job on every freebsd system:

 4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats

that matches up the rules with the directories, and every five minutes overwrites the users' /jc_traffic_dump with the latest traffic stats.