Editing FreeBSD Reference (section)

== Firewall Rule Configuration ==

The firewall startup script is found here:

 /etc/firewall.sh

It is created periodically based on the current ruleset.

The only thing we do with ipfw on the firewall is block or accept packets and occasionally cap some ips (we do not do any counting, or accounting).  

The first rule is to allow traffic pointed at the firewall itself to pass – this is to facilitate access in the event of a DoS attack. 
 00001 allow ip from any to 69.55.230.1

Rules 2-10 are for bandwidth capping and blocking bad people:

 00002 pipe 2 ip from 69.55.224.109 to any xmit em0
 00003 pipe 3 ip from { 69.55.227.54 or 69.55.227.55 } to any xmit em0
 00004 pipe 4 ip from 69.55.238.194 to any xmit em0
 00005 pipe 5 ip from 69.55.238.162 to any xmit em0
 00006 deny ip from 69.22.167.138 to any

Rule 100 is for our infrastructure machines:

 00100 allow udp from any 53 to 69.55.230.2
 00100 allow udp from 69.55.230.2 123 to any
 00100 allow udp from 69.55.230.2 to any dst-port 53
 00100 allow tcp from any to 69.55.230.2 dst-port 22,25,80,443,110,123,1984,8080 setup
 00100 allow icmp from any to 69.55.230.2 icmptypes 0,3,8 keep-state
 00100 allow udp from 69.55.230.1 161 to 69.55.230.2
 00100 deny ip from any to 69.55.230.2
 00100 allow tcp from any to 65.55.238.150 dst-port 25 setup

Rules 101-150 are for jails/virts they disable all traffic from the pub net except from mail, backup, dns, and virtuozzo:
<pre>
00101 deny ip from any to 69.55.238.120
00102 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.53
00102 deny ip from any to 69.55.228.53
00103 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.64
00103 deny ip from any to 69.55.238.64
00104 allow ip from { 69.55.230.2 or 69.55.230.9 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.92
00104 deny ip from any to 69.55.238.92
00106 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.180
00106 deny ip from any to 69.55.238.180
00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.210
00107 deny ip from any to 69.55.238.210
00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.237.129
00109 deny ip from any to 69.55.237.129
00110 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.236.128
00110 deny ip from any to 69.55.236.128
00111 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.236.92
00111 deny ip from any to 69.55.236.92
00112 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.235.200
00112 deny ip from any to 69.55.235.200
00113 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.225.2
00113 deny ip from any to 69.55.225.2
00114 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.226.128
00114 deny ip from any to 69.55.226.128
00115 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.224.32
00115 deny ip from any to 69.55.224.32
00116 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.224.110
00116 deny ip from any to 69.55.224.110
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2
00130 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.2
00130 deny ip from any to 69.55.227.2
00132 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.237.220
00132 deny ip from any to 69.55.237.220
00133 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.236.192
00133 deny ip from any to 69.55.236.192
00134 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.236.64
00134 deny ip from any to 69.55.236.64
00135 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.235.170
00135 deny ip from any to 69.55.235.170
00136 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.234.151
00136 deny ip from any to 69.55.234.151
00137 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.225.77
00137 deny ip from any to 69.55.225.77
00138 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.226.2
00138 deny ip from any to 69.55.226.2
00139 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.226.161
00139 deny ip from any to 69.55.226.161
00140 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.224.150
00140 deny ip from any to 69.55.224.150
00141 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.70
00141 deny ip from any to 69.55.227.70
00141 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.229.2
00142 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.70
00142 deny ip from any to 69.55.227.70
00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.230.18
00143 deny ip from any to 69.55.230.18
00144 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.229.100
00144 deny ip from any to 69.55.229.100</pre>
In addition to rule 00010 (allow all established) and rule 65500 (allow all) we also have a few more special rules:

<pre>00012 deny tcp from any to any tcpflags syn tcpoptions !mss
00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
00012 deny tcp from any to any tcpflags syn,fin
00012 deny tcp from any to any tcpflags fin,psh,rst,urg
00012 allow icmp from any to any
00013 allow udp from any to 69.55.225.225 dst-port 53
00014 deny tcp from any to any dst-port 135</pre>

These are the four DoS attack lines we have in place right at the beginning of the ruleset.

When the machine boots, and is running only three total rules, you then log in and run /etc/firewall.sh - that contains all the additional rules - running the script will put them all in place - then the firewall is fully configured.

Now, by default, we do not put any rules in place at all for a customer - they are left wide open.  Most customers do not ever change this. However, if a customer requests a ruleset on our firewall, we implement it in the general form that was described above - allow all ports that need to be open, and deny all others.

The firewall rule numbers are not numbered arbitrarily - they are numbered by customer number.  So customer 327 gets 03270 - 03279, and customer 589 gets 05890 - 05899 ... once we get to customer 1000, they will have 10000 - 10009.
 
This does not mean that every customer can only have 10 rules - as you can see from the four DoS attack rules that are all numbered 00003, you can create multiple rules at the same rule number.  I don't advise it though.

Because customer requests are generally "allow these and block everything else" we actually have a script on the firewall to create a typical ruleset.  The script is called "rulemaker", and it runs like this:

 # rulemaker
 usage:  rulemaker [cust#] IP [port1,port2,...,port10]

So, it has three command line options - the customer number (significant digits only), the IP, and a comma-delimited list of ports (with no spaces).

So, if customer 398 comes to you and says:

"please open up tcp ports for ssh, smtp and http and close all the rest"

then you would run:

 rulemaker 398 10.10.10.10 22,25,80

And this is what would happen:

<pre>gateway# rulemaker 398 10.10.10.10 22,25,80
/sbin/ipfw add 03981 allow udp from 10.10.10.10 to any 53
/sbin/ipfw add 03982 allow udp from any 53 to 10.10.10.10
/sbin/ipfw add 03983 allow tcp from any to 10.10.10.10 22,25,80 setup
/sbin/ipfw add 03989 deny ip from any to 10.10.10.10

or, if they have a dns server:

/sbin/ipfw add 03981 allow udp from 10.10.10.10 to any 53
/sbin/ipfw add 03982 allow udp from any 53 to 10.10.10.10
/sbin/ipfw add 03983 allow tcp from any to 10.10.10.10 22,25,53,80 setup
/sbin/ipfw add 03984 allow udp from any to 10.10.10.10 53
/sbin/ipfw add 03989 deny ip from any to 10.10.10.10

REMEMBER TO ADD YOUR PASTE TO /usr/local/etc/ipfw.sh
gateway#</pre>
if they have dns, put a 53 in the command line arg to rulemaker

You are shown a list of rules to paste into place if they don't run a dns server, and one if they do.

Note that rulemaker does not actually put any rules in place at all, it just echos the commands you should run.  So, since the customer did not specify port 53, we can assume they do not run a dns server, and we can simply paste this:
<pre>
/sbin/ipfw add 03981 allow udp from 10.10.10.10 to any 53
/sbin/ipfw add 03982 allow udp from any 53 to 10.10.10.10
/sbin/ipfw add 03983 allow tcp from any to 10.10.10.10 22,25,80 setup
/sbin/ipfw add 03989 deny ip from any to 10.10.10.10</pre>
into the shell, and hit enter once or twice afterwards. Very simple.  We then email the customer and tell them that the lines are in place, and to test them.

customer numbers larger than 999 will work fine with this script because:

 ipfw add 010000 (rule)

and

 ipfw add 10000 (rule)

translate into the same thing.  So adding unnecessary zeroes does not hurt anything.  (the rulemaker script outputs 0$1 as the rule number - so it always prepends a zero to make the three-digit customer numbers correct, and that zero prepended to a four digit customer number will not hurt anything - it will just be ignored)

Almost every rule in the firewall is part of a little 4 or 5-line set like rulemaker outputs.  Some exceptions are when people want you to open up icmp for them as well (since the above rulemaker output denies it) in which case you would simply paste the rulemaker output, and then afterwards add another rule:

 ipfw add 03984 allow icmp from any to 10.10.10.10

Remember, if they run a dns server, they need to have tcp port 53 in their port list and you need to paste the second block that rulemaker outputs.

Some customers, however, do not request a formal ruleset - they simply say to block off port 3306 from the outside (mysql) or they say to block all netbios ports (135,137,139) or something like that.  If they do this, do not use rulemaker - simply add a rule just for that:

 /sbin/ipfw add 05431 deny tcp from any to 10.10.10.10 3306

or

 /sbin/ipfw add 05431 deny tcp from any to 10.10.10.10 135,137,139

On the other hand, a customer may request a normal ruleset, but then request that you only open ssh for a certain IP block or IP.  Here is an example of a ruleset that was started with rulemaker, but then additional rules were added:
<pre>
07471      47802    3991038 allow udp from 69.55.225.125 to any 53
07472      14490    1309166 allow udp from any 53 to 69.55.225.125
07473      85950    4252824 allow tcp from any to 69.55.225.125 22,25,53,80,443,110,143,220 setup
07474      45358    3378454 allow udp from any to 69.55.225.125 53
07475         84       5016 allow tcp from any to 69.55.225.127 22,443
07475         94       5472 allow tcp from any to 69.55.225.128 22,443
07476      38805    3552124 allow icmp from any to 69.55.225.127
07476      38524    3536996 allow icmp from any to 69.55.225.128
07478          6        288 allow tcp from 66.166.221.232/29 to 69.55.225.125 3309
07478        286      13728 allow tcp from 66.166.221.232/29 to 69.55.225.125 3306
07479     109767    6222136 deny ip from any to { 69.55.225.125 or dst-ip 69.55.225.127 or dst-ip 69.55.225.128 }</pre>

So ... 69.55.225.125 is the main IP, and what was used in rulemaker, and the main allow line is very familiar:

 07473      85950    4252824 allow tcp from any to 69.55.225.125 22,25,53,80,443,110,143,220 setup

but then they wanted allow only 22 and 443 to the other two IP addresses:

 07475         84       5016 allow tcp from any to 69.55.225.127 22,443
 07475         94       5472 allow tcp from any to 69.55.225.128 22,443

(note they share an ipfw rule number)

then icmp should also be allowed to the other two IPs:

 07476      38805    3552124 allow icmp from any to 69.55.225.127
 07476      38524    3536996 allow icmp from any to 69.55.225.128

then there are two addresses out in the world that should be totally unfettered in their ability to talk to the main IP:

 07478          6        288 allow tcp from 66.166.221.232/29 to 69.55.225.125

or to two ports

 07478        286      13728 allow tcp from 66.166.221.232/29 to 69.55.225.125 3306
 07478        286      13728 allow tcp from 66.166.221.232/29 to 69.55.225.125 3309

(note, again, sharing ipfw numbers, and also specifying a netblock instead of a single IP:  66.166.221.232/29)

then finally, the last rule that rulemaker outputs was thrown out and this was used instead:

 07479     109767    6222136 deny ip from any to { 69.55.225.125 or dst-ip 69.55.225.127 or dst-ip 69.55.225.128 }

Since we are dealing with three IPs total.

Some more example requests:

Replacing a rule (customer wants port 21 access):
<pre>gateway# g 69.55.225.3
07161      22462    1795170 allow udp from 69.55.225.3 to any dst-port 53
07162      21220    3283214 allow udp from any 53 to 69.55.225.3
07163      52962    2989600 allow tcp from any to 69.55.225.3 dst-port 22,80,443,25,110,995,143,993,53 setup
07164      20234    1314826 allow udp from any to 69.55.225.3 dst-port 53
07169      30715    2409544 deny ip from any to 69.55.225.3
gateway#
gateway# ipfw del 07163 ; ipfw add 07163 allow tcp from any to 69.55.225.3 20,21,22,80,443,25,110,995,143,993,53 setup
07163 allow tcp from any to 69.55.225.3 20,21,22,80,443,25,110,995,143,993,53 setup
gateway#</pre>

Please block all traffic from this range of IPs:
Inet num:      195.238.48.0 - 195.238.63.255

<pre>gateway# g 69.55.226.144
08441        356      21668 allow udp from 69.55.226.144 to any dst-port 53
08442       6744    1114132 allow udp from any 53 to 69.55.226.144
08443       7358     411368 allow tcp from any to 69.55.226.144 dst-port 22,25,80,110,443 setup
08449       3135     280030 deny ip from any to 69.55.226.144
gateway#
gateway# ipfw add 08440 deny ip from 195.238.48.0/20 to 69.55.226.144</pre>
in reply, say “your ruleset is now…”

/etc/firewall.sh is backed up daily locally (/etc/oldrules) and to the backup server

----------------

We add rules to block traffic from directly contacting our jails/virts.
Each rule is basically the same except for the id (which reflects the machine) and the machine’s IP
Here’s some examples:

Jail2:
 00102 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.2
 00102 deny ip from any to 69.55.238.2

Quar1:
 00130 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.2
 00130 deny ip from any to 69.55.227.2

Virt12:
 00142 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.229.2
 00142 deny ip from any to 69.55.229.2

The IPs listed for access are mail (the new mail), backup2, ns1c, and virtuozzo

To dump/watch traffic:

 tcpdump –vvv –n –i em1