FreeBSD Reference
Dump and Restore
Last updated 2005-04-12
One of the basic tasks that we need to perform in this organization is creating filesystem images and accurately cloning them across servers.
Since each server instance running on the machines has a full FreeBSD filesystem, it's important to understand the correct ways of cloning a filesystem and extracting the filesystem into multiple places.
The natural assumption is to use tar(1). However, tar(1) is not appropriate for archiving special files and for preserving all ownership and special attributes of certain files. This is important to keep in mind, because when we create a file system image, we recreate the entire OS. Run this command to see what type of "special" files are sometimes not treated correctly with tar(1):
`ls -asl /dev | more`
Please note the major/minor device numbers that these files have.
It would be difficult to describe all the shortcomings of tar(1), but they exist, and must be avoided by using different tools. An almost monthly question that appears on the FreeBSD mailing lists is "what should I use to safely backup this kind of filesystem or these kind of files and be sure that I am faithfully preserving everything." This is where dump(8) and restore(8) come in. A FreeBSD core member was once quoted on one of the mailing lists saying "Use dump or you will lose".
It is actually unfortunate that we cannot use tar(1) because it is much simpler, and more versatile. Further, many of the files that we are guaranteeing accuracy on by using dump(8) and restore(8) are probably not that important in the environment we run he server instances in. We provide a full /dev to our customers, but really they need very little of it. However, it's not for me to say why they are getting a FreeBSD server, and so I try to reproduce a real one as faithfully as possible.
Basically the dump(8) command dumps a filesystem to a regular file, and the restore(8) command restores the data from that file into a path you specify at some later date. dump(8) is non-versatile in the sense that you can only dump filesystems. So, you can dump /var (if you have /var as a seperate partition), but you cannot dump /var/db by itself, since presumably that is not on its own partition. If you want to dump /var/db, you need to dump all of /var. And if /var was not its own partition, you would have to dump all of / to get it. This is unfortunate, but it is how dump(8) works and there is no getting around it.
restore(8) is more flexible in the sense that you can restore a dump-file into any location. Just cd to where you want the dump-file expanded and expand it. Easy.
Let's give it a shot:
First, I run the command `df -k` to see what filesystems I have on my server:
www# df -k Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/mlxd0s1a 1016303 862925 72074 92% / /dev/mlxd0s1f 7030220 5488394 979409 85% /mnt/data1 /dev/mlxd1s1e 17369623 11692971 4287083 73% /mnt/data2 /dev/mlxd0s1e 508143 181167 286325 39% /var procfs 4 4 0 100% /proc www#
I'll choose the /var partition for this example since it is small and will go quickly. The syntax for dumping a filesystem to a file is:
`dump -0a -f /mnt/data2/dump-file /dev/mlxd0s1e`
So, we are dumping to a file /mnt/data2/dump-file that does not already exist, and we are dumping from the device /dev/mlxd0s1e which, as you can see from the df(1) output is the device that /var is mounted on. It is worth repeating that you can only dump an entire filesystem, and that filesystem is referred to by the device it is mounted on.
www# dump -0a -f /mnt/data2/dump-file /dev/mlxd0s1e DUMP: Date of this level 0 dump: Tue Aug 20 00:21:25 2002 DUMP: Date of last level 0 dump: the epoch DUMP: Dumping /dev/mlxd0s1e (/var) to /mnt/data2/dump-file DUMP: mapping (Pass I) [regular files] DUMP: mapping (Pass II) [directories] DUMP: estimated 181700 tape blocks. DUMP: dumping (Pass III) [directories] DUMP: dumping (Pass IV) [regular files] DUMP: DUMP: 181684 tape blocks on 1 volume DUMP: finished in 52 seconds, throughput 3493 KBytes/sec DUMP: Closing /mnt/data2/dump-file DUMP: DUMP IS DONE www#
So now we have a backup of /var in a dump-file named /mntdata2/dump-file. We can just save that until we need it - it is ok to compress it or tar it up or even rename it - it is simply a regular file.
Now we wish to restore the contents of dump-file using the restore(8) command. Remember that we can restore the file anywhere – we can restore it on a partition like we dumped from (such as /var) or we can restore it in a deeply nested directory such as /usr/local/etc/rc.d. It will restore correctly regardless of where you restore it.
www# www# mkdir /mnt/data2/test www# mkdir /mnt/data2/test/test www# cd /mnt/data2/test/test www# restore -x -f /mnt/data2/dump-file
You have not read any tapes yet. Unless you know which volume your file(s) are on you should start with the last volume and work towards the first. Specify next volume #: 1 set owner/mode for '.'? [yn] y www#
These are probably the only arguments you will ever use for restore(8) unless you place it in a script which we will discuss in a bit. You'll note that two pieces of user input are needed in this process, as opposed to dump(8) which did not ask any questions. First, we are asked to "Specify next volume #:", to which we reply "1" (and we will almost certainly always simply reply with "1", since we are not doing multi volume tape sets or anything complicated lke that). Second, many seconds later after the dump is complete, we are asked if we want to "set owner/mode for '.'?" and the answer is Yes, or more specifically, the y key followed by a carriage return. The whole point of using these tools is to preserve attributes, so it is natural that we would say yes to this question.
Now that you’ve learned how to do it manually, you’ll be happy to know there’s a script that automatically enters the 1 and y for you: dumprestore <path-to-dump-file>
however…this sometimes fails to set proper permissions. So either followup or don’t use it.
Some additional information that you should know:
First, dump(8) and restore(8) are not just FreeBSD commands. They are very old, historical unix commands. They have the ability to interact with complex systems of tapes/volumes and are the core of a lot of enterprise backup schemes. You would be surprised how many unix admins use these tools instead of big fancy backup products.
Second, whereas the only syntax we will probably ever use with dump(8) is this:
dump -0a -f </target/file> </device/name>
there is another syntax that we will sometimes use with restore(8). In our example, we restored in an interactive environment - that is, we are logged onto a server and running restore(8) in a shell. We use this syntax:
restore -x -f /some/dump-file
and we then answer the two questions that require an interactive response. However, sometimes we will want to run restore(8) out of a script, which presents a problem because we cannot answer the questions in a script when it runs. Therefore, if we restore a dump-file as part of a shell script, we use this syntax instead:
`restore -rf /some/dump-file`
This is just slightly different - it performs the restore without needing any interaction. However, it also leaves behind a file called `restoresymtable` in the directory where you restore the dump-file. This file can be safely deleted.
One other consideration that is worth noting is that dump(8) does not save mount points of other partitions. If you dump / and /var is a separate partition, when you restore the dump-file, you will not have a /var directory in the directory where you restore the dump-file.
So now you should try these things out on your own. Install FreeBSD on a system and dump the entire / filesystem into a file. Strange as it may seem, you can actually dump / into /. For instance, if /data is not its own partition and is simply a directory under /, you can dump / into /data/dump-file.
After dumping / into a dump-file, dump /var into a file called dump-file-var. Now create any old directory and restore the dump-file. Then create a /var directory inside the target where you restored your dump-file, cd to it, and restore the dump-file-var file.
Presuming you had no other meaningful partitions besides /var that were on their own device, you now have a clone of your entire system inside /data (or wherever you restored things).
ipfw (firewalling)
ipfw is a userland command (/sbin/ipfw) that is, in a sense, the FreeBSD equivalent of the linux `iptables` command. Although the ipfw command is present in all FreeBSD installations, it also requires kernel support.
You can enable ipfw by adding this line to your kernel configuration line:
options IPFIREWALL
and building and installing that new kernel. It should be noted, however, that there are two other options that should probably be added to your kernel configuration file as well:
options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=100
The first allows ipfw to do verbose logging of packets and events, and the second limits the number of log entries that a single, consecutive event can log.
ipfw support can also be loaded as a module, by simply running:
/sbin/kldload ipfw.ko
It is VERY IMPORTANT to understand that ipfw, when existent in a kernel, or when loaded as a kernel module with kldload, will _always_ have a single default, final rule. That is, there will always be at least one rule present, and that rule will be the final, catch-all rule for any packet that does not match any prior rules.
By default, the final, default rules is:
deny ip from any to any
Which means (VERY IMPORTANT) that if you enable the firewall in your kernel and then reboot, you will have no network connectivity to the machine in question _at all_. The machine will have a single rule, which is set to deny all ip traffic, and you will not be able to reach it in any way.
Similarly, if you simply run `/sbin/kldload ipfw.ko` on a normal FreeBSD system, you will be instantly locked out of the system, and the system will be completely unreachable from the network.
There are a few ways to solve this problem. First, you can add a fourth line to your kernel configuration file:
options IPFIREWALL_DEFAULT_TO_ACCEPT
This means that ipfw support will be added to the kernel, but with a final, default rule of:
allow ip from any to any
With this rule in place as the final, default rule, even if you add no other rules, when the system restarts you will be able to see it on the network just fine, and no packets will be blocked.
Another way to solve the problem is to leave ipfw in the kernel set to the default (which is to deny ip from any to any) but a rule to your startup configuration to open up some or all network services. The default will still be to deny ip from any to any, but any traffic the first matches your allow rules will be allowed in. This method will work fine, regardless of whether you added ipfw support in the kernel or by loading a module. In fact, you can (theoretically, although I distrust this method) run this command:
/sbin/kldload ipfw.ko ; ipfw add 65500 allow ip from any to any
and not be locked out ... the default rule is always number 65535, so by instantly adding a rule at 65500 to allow all traffic, you can load the ipfw module safely without locking yourself out. I do not recommend this, however, as I do not trust that this will work 100% of the time.
Finally, a third method to enable ipfw without locking yourself out is to rebuild the ipfw.ko module to set its default to allow, instead of deny - just like we were able to set a kernel configuration line to set the default to allow. Building this module with that custom configuration is beyond the scope of this document.
Why does FreeBSD have a default setting for ipfw to deny all traffic ? The reason is, you do not want to allow a malicious party to circumvent your firewall rules by crashing your machine. In some configurations, firewall rules may not be loaded at boot time, so if the default was "allow all from any to any" then one could circumvent the firewall rules by crashing the firewall - when it came back up the rules would not be loaded, and all traffic would pass.
REMEMBER - no matter what you do, there will always be a single rule in place - rule number 65535. You cannot delete this rule, and it is set to deny all or allow all depending on how it was set (as we discussed above).
You can view all of the rules currently active on a system by running:
/sbin/ipfw show
A useful way to see all the rules that might apply to a certain IP, for instance 10.10.10.10, is:
/sbin/ipfw show | grep "10.10.10.10"
Here is what the results of `ipfw show` would look like on a system with a default accept configuration:
# ipfw show 65535 109490 44385056 allow ip from any to any
As you can see, there is only one rule - number 65535. You can only configure 65535 rules, and that is always the last one, and is always present.
The second number in the output is the packet count - how many packets have passed through that rule, and the third number is the byte count - how many bytes have passed through that rule.
You add rules with commands like:
ipfw add 100 allow tcp from any to 10.10.10.10 22
That line allows all tcp traffic on port 22 destined for 10.10.10.10.
If you now run ipfw show, you would see:
# ipfw show 00100 0 0 allow tcp from any to 10.10.10.10 22 65535 116103 46464923 allow ip from any to any
You could then delete rule #100 with:
ipfw del 00100
A very useful and command setup for a system behind a firewall is to open up only the ports that correspond to services actually running on that system, and deny all other traffic. Here is an example, where the IP in question is 10.10.10.10:
ipfw add 100 allow tcp from any to 10.10.10.10 established ipfw add 200 allow tcp from any to 10.10.10.10 22,25,80,443 setup ipfw add 300 deny tcp from any to 10.10.10.10
So, in this example, we first allow any previously established tcp connections to this IP, then we allow any tcp connections that are in a setup (TCP 3-way handshake) mode - these two rules together account for all possible legitimate tcp traffic. Then the third rule simply denies all other tcp traffic.
As you can see, ipfw applies rules FIFO - so as a packet travels from rule 0 to rule 65535, as soon as it matches a rule, the packet is processed, and leaves the ruleset - no further rules are applied to that packet, as it has been passed.
In the above example, the user is running ssh, smtp, http, and https ... however, for a system to be workable on the network, a few other things should be open if they run a dns server - since dns uses UDP as well:
ipfw add 100 allow tcp from any to 10.10.10.10 established ipfw add 200 allow tcp from any to 10.10.10.10 22,25,53,80,443 setup ipfw add 300 allow udp from any to 10.10.10.10 53 ipfw add 400 deny ip from any to 10.10.10.10
Note the addition of rule 300, which allows udp to come in on port 53. Also notice that since dns uses both tcp and udp, we have added 53 to the allowed tcp list as well. Finally, note that we can refer to ipfw rule numbers as either 00200 or 200 - it is the same thing.
Also, note that the final rule for this IP, #400, does not simply deny all tcp or all udp (or both), rather, it denies all IP _period_. So if it doesn't match the tcp list, and if it is not udp53, then it gets dropped as soon as it hits rule 400.
It should be noted that there is performance penalty on the firewall machine for each rule added to it _that packets pass through_. (so, even if you have 1000 rules on the firewall, if the first rule is `allow ip from any to any` then there is no difference in performance than if the ruleset had only one rule - all packets get passed right away and none pass trough the other 999 rules)
If a poor ruleset design is in use, and the firewall takes in a lot of traffic (or passes a DoS attack) it can knock the firewall off the network. The machine will not be crashed - as soon as the traffic lets up it will respond again - but no communication will pass while the CPU is overloaded.
The number one consideration when tuning the ruleset for performance is to pass packets that should be passed as fast as possible, and dump packets that should be dumped as fast as possible. For example, we know that we don't want to ever pass certain types of packets (a tcp packet with all option fields set, a tcp packet with no MSS setting, and so on) so, the very first four rules on our firewall are:
00003 49913883 2009604713 deny tcp from any to any tcpflags syn tcpoptions !mss 00003 23958169 1342587681 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18 00003 142 8496 deny tcp from any to any tcpflags syn,fin 00003 0 0 deny tcp from any to any tcpflags fin,psh,rst,urg
This means that these packets, if we ever encounter them (and we frequently do in a DoS attack) are dropped immediately. If we had 1000 rules in our ruleset, and put these at the end, a big DoS attack would cripple the firewall, because _every single_ other rule would have to be processed against every single packet until it got spit out the end.
On the other side of the coin, we also allow what we know should be allowed _as fast as we can_.
Take the example above:
ipfw add 100 allow tcp from any to 10.10.10.10 established ipfw add 200 allow tcp from any to 10.10.10.10 22,25,80,443 setup ipfw add 300 deny tcp from any to 10.10.10.10
Now, lets say we have four machines on our network, so we set up a block of rules for all four:
ipfw add 100 allow tcp from any to 10.10.10.10 established ipfw add 150 allow tcp from any to 10.10.10.10 22,25,80,443 setup ipfw add 200 deny tcp from any to 10.10.10.10 ipfw add 250 allow tcp from any to 10.10.10.20 established ipfw add 300 allow tcp from any to 10.10.10.20 22,25,80,443 setup ipfw add 350 deny tcp from any to 10.10.10.20 ipfw add 400 allow tcp from any to 10.10.10.30 established ipfw add 450 allow tcp from any to 10.10.10.30 22,25,80,443 setup ipfw add 500 deny tcp from any to 10.10.10.30 ipfw add 550 allow tcp from any to 10.10.10.40 established ipfw add 600 allow tcp from any to 10.10.10.40 22,25,80,443 setup ipfw add 650 deny tcp from any to 10.10.10.40
Now, each ruleset applies to a particular IP. The problem is, we can tell by looking at this that we _always_ pass established tcp packets – no matter what IP they are for. If an established packet comes in for 10.10.10.40, it first has to pass through 9 other rules before it gets to:
ipfw add 550 allow tcp from any to 10.10.10.40 established
Since we know we always pass established packets, we can shorten this dramatically by doing this:
ipfw add 001 allow tcp from any to any established ipfw add 150 allow tcp from any to 10.10.10.10 22,25,80,443 setup ipfw add 200 deny tcp from any to 10.10.10.10 ipfw add 300 allow tcp from any to 10.10.10.20 22,25,80,443 setup ipfw add 350 deny tcp from any to 10.10.10.20 ipfw add 450 allow tcp from any to 10.10.10.30 22,25,80,443 setup ipfw add 500 deny tcp from any to 10.10.10.30 ipfw add 600 allow tcp from any to 10.10.10.40 22,25,80,443 setup ipfw add 650 deny tcp from any to 10.10.10.40
Now an established tcp connection (that we always want to pass) is passed at rule #1 - as fast as it possible can be passed. If we have hundreds of systems behind this firewall, with rulesets such as this, this can cause the CPU of the firewall to be totally idle, vs. being 50% utilized – just by putting that change in. I have seen it. This is true because a large percentage of all traffic at all times is established tcp traffic.
Also, note that we always deny ip from any to (the IP) as soon as that IPs ruleset is over - this is not technically necessary - you could just let the final 65535 "deny all" rule catch it and deny it there - but that would kill performance because the packet would have to be matched against every other rule inbetween first.
If you forget to put in index #, it will add as 65535
Firewall Rule Configuration
The firewall startup script is found here:
/etc/firewall.sh
It is created periodically based on the current ruleset.
The only thing we do with ipfw on the firewall is block or accept packets and occasionally cap some ips (we do not do any counting, or accounting).
The first rule is to allow traffic pointed at the firewall itself to pass – this is to facilitate access in the event of a DoS attack.
00001 allow ip from any to 69.55.230.1
Rules 2-10 are for bandwidth capping and blocking bad people:
00002 pipe 2 ip from 69.55.224.109 to any xmit em0
00003 pipe 3 ip from { 69.55.227.54 or 69.55.227.55 } to any xmit em0
00004 pipe 4 ip from 69.55.238.194 to any xmit em0
00005 pipe 5 ip from 69.55.238.162 to any xmit em0
00006 deny ip from 69.22.167.138 to any
Rule 100 is for our infrastructure machines:
00100 allow udp from any 53 to 69.55.230.2 00100 allow udp from 69.55.230.2 123 to any 00100 allow udp from 69.55.230.2 to any dst-port 53 00100 allow tcp from any to 69.55.230.2 dst-port 22,25,80,443,110,123,1984,8080 setup 00100 allow icmp from any to 69.55.230.2 icmptypes 0,3,8 keep-state 00100 allow udp from 69.55.230.1 161 to 69.55.230.2 00100 deny ip from any to 69.55.230.2 00100 allow tcp from any to 65.55.238.150 dst-port 25 setup
Rules 101-150 are for jails/virts they disable all traffic from the pub net except from mail, backup, dns, and virtuozzo:
00101 deny ip from any to 69.55.238.120
00102 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.53
00102 deny ip from any to 69.55.228.53
00103 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.64
00103 deny ip from any to 69.55.238.64
00104 allow ip from { 69.55.230.2 or 69.55.230.9 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.92
00104 deny ip from any to 69.55.238.92
00106 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.180
00106 deny ip from any to 69.55.238.180
00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.210
00107 deny ip from any to 69.55.238.210
00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.237.129
00109 deny ip from any to 69.55.237.129
00110 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.236.128
00110 deny ip from any to 69.55.236.128
00111 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.236.92
00111 deny ip from any to 69.55.236.92
00112 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.235.200
00112 deny ip from any to 69.55.235.200
00113 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.225.2
00113 deny ip from any to 69.55.225.2
00114 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.226.128
00114 deny ip from any to 69.55.226.128
00115 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.224.32
00115 deny ip from any to 69.55.224.32
00116 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.224.110
00116 deny ip from any to 69.55.224.110
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2
00130 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.2
00130 deny ip from any to 69.55.227.2
00132 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.237.220
00132 deny ip from any to 69.55.237.220
00133 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.236.192
00133 deny ip from any to 69.55.236.192
00134 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.236.64
00134 deny ip from any to 69.55.236.64
00135 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.235.170
00135 deny ip from any to 69.55.235.170
00136 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.234.151
00136 deny ip from any to 69.55.234.151
00137 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.225.77
00137 deny ip from any to 69.55.225.77
00138 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.226.2
00138 deny ip from any to 69.55.226.2
00139 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.226.161
00139 deny ip from any to 69.55.226.161
00140 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.224.150
00140 deny ip from any to 69.55.224.150
00141 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.70
00141 deny ip from any to 69.55.227.70
00141 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.229.2
00142 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.70
00142 deny ip from any to 69.55.227.70
00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.230.18
00143 deny ip from any to 69.55.230.18
00144 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.229.100
00144 deny ip from any to 69.55.229.100
In addition to rule 00010 (allow all established) and rule 65500 (allow all) we also have a few more special rules:
00012 deny tcp from any to any tcpflags syn tcpoptions !mss 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18 00012 deny tcp from any to any tcpflags syn,fin 00012 deny tcp from any to any tcpflags fin,psh,rst,urg 00012 allow icmp from any to any 00013 allow udp from any to 69.55.225.225 dst-port 53 00014 deny tcp from any to any dst-port 135
These are the four DoS attack lines we have in place right at the beginning of the ruleset.
When the machine boots, and is running only three total rules, you then log in and run /etc/firewall.sh - that contains all the additional rules - running the script will put them all in place - then the firewall is fully configured.
Now, by default, we do not put any rules in place at all for a customer - they are left wide open. Most customers do not ever change this. However, if a customer requests a ruleset on our firewall, we implement it in the general form that was described above - allow all ports that need to be open, and deny all others.
The firewall rule numbers are not numbered arbitrarily - they are numbered by customer number. So customer 327 gets 03270 - 03279, and customer 589 gets 05890 - 05899 ... once we get to customer 1000, they will have 10000 - 10009.
This does not mean that every customer can only have 10 rules - as you can see from the four DoS attack rules that are all numbered 00003, you can create multiple rules at the same rule number. I don't advise it though.
Because customer requests are generally "allow these and block everything else" we actually have a script on the firewall to create a typical ruleset. The script is called "rulemaker", and it runs like this:
# rulemaker usage: rulemaker [cust#] IP [port1,port2,...,port10]
So, it has three command line options - the customer number (significant digits only), the IP, and a comma-delimited list of ports (with no spaces).
So, if customer 398 comes to you and says:
"please open up tcp ports for ssh, smtp and http and close all the rest"
then you would run:
rulemaker 398 10.10.10.10 22,25,80
And this is what would happen:
gateway# rulemaker 398 10.10.10.10 22,25,80 /sbin/ipfw add 03981 allow udp from 10.10.10.10 to any 53 /sbin/ipfw add 03982 allow udp from any 53 to 10.10.10.10 /sbin/ipfw add 03983 allow tcp from any to 10.10.10.10 22,25,80 setup /sbin/ipfw add 03989 deny ip from any to 10.10.10.10 or, if they have a dns server: /sbin/ipfw add 03981 allow udp from 10.10.10.10 to any 53 /sbin/ipfw add 03982 allow udp from any 53 to 10.10.10.10 /sbin/ipfw add 03983 allow tcp from any to 10.10.10.10 22,25,53,80 setup /sbin/ipfw add 03984 allow udp from any to 10.10.10.10 53 /sbin/ipfw add 03989 deny ip from any to 10.10.10.10 REMEMBER TO ADD YOUR PASTE TO /usr/local/etc/ipfw.sh gateway#
if they have dns, put a 53 in the command line arg to rulemaker
You are shown a list of rules to paste into place if they don't run a dns server, and one if they do.
Note that rulemaker does not actually put any rules in place at all, it just echos the commands you should run. So, since the customer did not specify port 53, we can assume they do not run a dns server, and we can simply paste this:
/sbin/ipfw add 03981 allow udp from 10.10.10.10 to any 53 /sbin/ipfw add 03982 allow udp from any 53 to 10.10.10.10 /sbin/ipfw add 03983 allow tcp from any to 10.10.10.10 22,25,80 setup /sbin/ipfw add 03989 deny ip from any to 10.10.10.10
into the shell, and hit enter once or twice afterwards. Very simple. We then email the customer and tell them that the lines are in place, and to test them.
customer numbers larger than 999 will work fine with this script because:
ipfw add 010000 (rule)
and
ipfw add 10000 (rule)
translate into the same thing. So adding unnecessary zeroes does not hurt anything. (the rulemaker script outputs 0$1 as the rule number - so it always prepends a zero to make the three-digit customer numbers correct, and that zero prepended to a four digit customer number will not hurt anything - it will just be ignored)
Almost every rule in the firewall is part of a little 4 or 5-line set like rulemaker outputs. Some exceptions are when people want you to open up icmp for them as well (since the above rulemaker output denies it) in which case you would simply paste the rulemaker output, and then afterwards add another rule:
ipfw add 03984 allow icmp from any to 10.10.10.10
Remember, if they run a dns server, they need to have tcp port 53 in their port list and you need to paste the second block that rulemaker outputs.
Some customers, however, do not request a formal ruleset - they simply say to block off port 3306 from the outside (mysql) or they say to block all netbios ports (135,137,139) or something like that. If they do this, do not use rulemaker - simply add a rule just for that:
/sbin/ipfw add 05431 deny tcp from any to 10.10.10.10 3306
or
/sbin/ipfw add 05431 deny tcp from any to 10.10.10.10 135,137,139
On the other hand, a customer may request a normal ruleset, but then request that you only open ssh for a certain IP block or IP. Here is an example of a ruleset that was started with rulemaker, but then additional rules were added:
07471 47802 3991038 allow udp from 69.55.225.125 to any 53
07472 14490 1309166 allow udp from any 53 to 69.55.225.125
07473 85950 4252824 allow tcp from any to 69.55.225.125 22,25,53,80,443,110,143,220 setup
07474 45358 3378454 allow udp from any to 69.55.225.125 53
07475 84 5016 allow tcp from any to 69.55.225.127 22,443
07475 94 5472 allow tcp from any to 69.55.225.128 22,443
07476 38805 3552124 allow icmp from any to 69.55.225.127
07476 38524 3536996 allow icmp from any to 69.55.225.128
07478 6 288 allow tcp from 66.166.221.232/29 to 69.55.225.125 3309
07478 286 13728 allow tcp from 66.166.221.232/29 to 69.55.225.125 3306
07479 109767 6222136 deny ip from any to { 69.55.225.125 or dst-ip 69.55.225.127 or dst-ip 69.55.225.128 }
So ... 69.55.225.125 is the main IP, and what was used in rulemaker, and the main allow line is very familiar:
07473 85950 4252824 allow tcp from any to 69.55.225.125 22,25,53,80,443,110,143,220 setup
but then they wanted allow only 22 and 443 to the other two IP addresses:
07475 84 5016 allow tcp from any to 69.55.225.127 22,443 07475 94 5472 allow tcp from any to 69.55.225.128 22,443
(note they share an ipfw rule number)
then icmp should also be allowed to the other two IPs:
07476 38805 3552124 allow icmp from any to 69.55.225.127 07476 38524 3536996 allow icmp from any to 69.55.225.128
then there are two addresses out in the world that should be totally unfettered in their ability to talk to the main IP:
07478 6 288 allow tcp from 66.166.221.232/29 to 69.55.225.125
or to two ports
07478 286 13728 allow tcp from 66.166.221.232/29 to 69.55.225.125 3306 07478 286 13728 allow tcp from 66.166.221.232/29 to 69.55.225.125 3309
(note, again, sharing ipfw numbers, and also specifying a netblock instead of a single IP: 66.166.221.232/29)
then finally, the last rule that rulemaker outputs was thrown out and this was used instead:
07479 109767 6222136 deny ip from any to { 69.55.225.125 or dst-ip 69.55.225.127 or dst-ip 69.55.225.128 }
Since we are dealing with three IPs total.
Some more example requests:
Replacing a rule (customer wants port 21 access):
gateway# g 69.55.225.3 07161 22462 1795170 allow udp from 69.55.225.3 to any dst-port 53 07162 21220 3283214 allow udp from any 53 to 69.55.225.3 07163 52962 2989600 allow tcp from any to 69.55.225.3 dst-port 22,80,443,25,110,995,143,993,53 setup 07164 20234 1314826 allow udp from any to 69.55.225.3 dst-port 53 07169 30715 2409544 deny ip from any to 69.55.225.3 gateway# gateway# ipfw del 07163 ; ipfw add 07163 allow tcp from any to 69.55.225.3 20,21,22,80,443,25,110,995,143,993,53 setup 07163 allow tcp from any to 69.55.225.3 20,21,22,80,443,25,110,995,143,993,53 setup gateway#
Please block all traffic from this range of IPs: Inet num: 195.238.48.0 - 195.238.63.255
gateway# g 69.55.226.144 08441 356 21668 allow udp from 69.55.226.144 to any dst-port 53 08442 6744 1114132 allow udp from any 53 to 69.55.226.144 08443 7358 411368 allow tcp from any to 69.55.226.144 dst-port 22,25,80,110,443 setup 08449 3135 280030 deny ip from any to 69.55.226.144 gateway# gateway# ipfw add 08440 deny ip from 195.238.48.0/20 to 69.55.226.144
in reply, say “your ruleset is now…”
/etc/firewall.sh is backed up daily locally (/etc/oldrules) and to the backup server
We add rules to block traffic from directly contacting our jails/virts. Each rule is basically the same except for the id (which reflects the machine) and the machine’s IP Here’s some examples:
Jail2:
00102 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.2
00102 deny ip from any to 69.55.238.2
Quar1:
00130 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.2
00130 deny ip from any to 69.55.227.2
Virt12:
00142 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.229.2
00142 deny ip from any to 69.55.229.2
The IPs listed for access are mail (the new mail), backup2, ns1c, and virtuozzo
To dump/watch traffic:
tcpdump –vvv –n –i em1
Setting up bandwidth caps
Creating a new pipe to limit someone's outbound speed: First make sure that you're not about to use a pipe that already exists.
newgateway# ipfw pipe list
00001: 1.000 Mbit/s 0 ms 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 tcp 69.55.224.109/44027 67.28.113.10/25 22 1320 0 0 0
newgateway#
there's already a pipe 1, so we'll use pipe 2, we're also going to add this as rule 2. (in this case the customer's IP is 69.55.224.109, and we only want to catch stuff going out so we use xmit em0.
newgateway# ipfw add 2 pipe 2 ip from 69.55.224.109 to any xmit em0 00002 pipe 2 ip from 69.55.224.109 to any xmit em0 newgateway#
Now all we have to do is set the speed limit:
newgateway# ipfw pipe 2 config bw 1Mbit/s newgateway#
Lastly, list the pipes to make sure everything is the way we want it:
newgateway# ipfw pipe list
00001: 1.000 Mbit/s 0 ms 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 tcp 69.55.224.109/44027 67.28.113.10/25 747 44980 0 0 0
00002: 1.000 Mbit/s 0 ms 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 tcp 69.55.224.109/80 62.172.72.131/26327 8468 9259344 40 42972 1038
newgateway#
Removing a pipe: the rule to match on and the pipe itself have to be deleted separately:
newgateway# ipfw delete 1 newgateway#
and to delete the pipe itself:
newgateway# ipfw pipe delete 1 newgateway#
list the pipes again:
newgateway# ipfw pipe show
00002: 1.000 Mbit/s 0 ms 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 tcp 69.55.224.109/80 62.172.72.131/26327 38383 42636953 48 50955 5111
newgateway#
more than one rule can feed into a pipe, so the speed of everything that matches will get lumped together in the same pipe. this is useful when a customer has more than on IP or system, and you want to limit his total combined speed.
ipfw on jail machines
The jail machines also have ipfw loaded, however all of the jail machines have ipfw loaded to default-accept. This is because ipfw on the jail machines is not used for firewalling - it is used for traffic counting. Note- ipfw will not run inside a jail- a jail vps customer cannot have its own self-managed ipfw rules. You see, in addition to deny and allow rules, you can also do things like:
ipfw add 00001 count ip from 10.10.10.10 to any ipfw add 00002 count ip from any to 10.10.10.10
which counts traffic bound from that IP, and bound to it. Where does it count it ? Simply in the ipfw rule itself. Here is a sample from one of the jail systems:
# ipfw show 00201 10 815 count ip from 198.78.70.176 to any 00202 10 802 count ip from any to 198.78.70.176 01631 1 62 count ip from 69.55.238.225 to any 01632 1 481 count ip from any to 69.55.238.225 01801 72 70154 count ip from 69.55.238.214 to any 01802 82 9047 count ip from any to 69.55.238.214 01811 3 245 count ip from 69.55.238.215 to any 01812 2 167 count ip from any to 69.55.238.215 01821 5 656 count ip from 198.78.66.216 to any 01822 4 377 count ip from any to 198.78.66.216 01841 0 0 count ip from 69.55.238.218 to any 01842 0 0 count ip from any to 69.55.238.218 01851 0 0 count ip from 198.78.66.219 to any 01852 0 0 count ip from any to 198.78.66.219 01861 3 218 count ip from 198.78.66.220 to any 01862 3 263 count ip from any to 198.78.66.220 01921 0 0 count ip from 69.55.238.224 to any 01922 0 0 count ip from any to 69.55.238.224 02241 0 0 count ip from 69.55.238.118 to any 02242 0 0 count ip from any to 69.55.238.118 02261 0 0 count ip from 69.55.238.223 to any 02262 0 0 count ip from any to 69.55.238.223 02271 0 0 count ip from 69.55.237.9 to any 02272 0 0 count ip from any to 69.55.237.9 02291 46 37023 count ip from 69.55.237.8 to any 02292 50 10939 count ip from any to 69.55.237.8 02311 20 1974 count ip from 69.55.237.7 to any 02312 22 1540 count ip from any to 69.55.237.7 03351 0 0 count ip from 69.55.237.163 to any 03352 0 0 count ip from any to 69.55.237.163 65535 102592563 113861945636 allow ip from any to any
Note two things - first, the packet counts and byte counts are very low, since this was taken shortly after system boot. Second, notice that the last line is "allow ip from any to any". That last line is the only line that affects actual traffic in any way - the others are just count rules.
Also, note that the rules are done by customer number - the customer number plus either a 1 or a 2 at the end - since every customer needs two total rules to count both inbound and outbound traffic. For instance:
01811 3 245 count ip from 69.55.238.215 to any 01812 2 167 count ip from any to 69.55.238.215
those are the two rules for customer 181 (col00181).
Remember how the `jailmake` utility asks for an "ipfw#" ? The three or four digit representation of the customer number is what it is asking for. As you can see, the last section of jailmake contains these lines:
/sbin/ipfw add `echo 0"$7"1` count ip from any to $ip /sbin/ipfw add `echo 0"$7"2` count ip from $ip to any
since the IP and the ipfw# are specified on the jailmake command line, this is very easy to do. Note again that the rule number is prepended with a zero - which does not hurt anything if it is an extra zero in the case of a four digit customer number.
The jails do not add each individual ipfw line at boot time like the firewall does. They are added by the postboot script which should only be run once to avoid duplicate ipfw entries.
The only odd thing about ipfw on the jails is that it is loaded as a module on jails 1-10, and loaded in the kernel in all jails 11 and beyond.
This is because doing traffic counting on the jails was not thought of until after we had loaded jail10. So, rather than schedule a maintenance and reboot all 10 jail systems, we simply built a default-allow module, placed it in the / directory, and loaded that at boot time. Therefore, on all jails 1-10, you will not only see something like this in the / directory:
ipfw.4.7.accept.ko
but you will also see a line like this in /usr/local/etc/rc.d/boot.sh:
/sbin/kldload /ipfw.4.7.accept.ko
(the module is named for the version of freebsd it was built for)
Jails 11-15 (and any future ones) have these lines:
options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=100 options IPFIREWALL_DEFAULT_TO_ACCEPT
in the kernel configuration file - note the default-accept line.
The traffic counting on the freebsd machines is not just for our benefit -to see by running `ipfw show` ... there is also a cron job on every freebsd system:
4,9,14,19,24,29,34,39,44,49,55,59 * * * * /usr/local/jail/bin/trafstats
that matches up the rules with the directories, and every five minutes overwrites the users' /jc_traffic_dump with the latest traffic stats.
Handling a DoS attack
When any attack occurs, usually Castle will catch it, stop it by null routing the IP. Sometimes our internal doswatch script will catch a UDP flood (DoS) so you’ll be aware at about the same time Castle is that there’s a DoS attack in progress. Other times you’ll have no idea an attack has happened until you get an email from Castle that there was a DoS attack- and sometimes they don’t send it “EMERGENCY” style. When you do get notification that there was a DoS attack, and if a null route has been placed, it’s important to contact the customer ASAP to 1) Notify him of what’s happened and, 2) Give him a new IP
Tasks:
- Create an entry in the Mgmt -> Reference -> DoSLog, using info fed to us by Castle (add the duration of the attack to the time of the attack to determine the time the attack ended). If he’s receiving a new IP (see below), add a note about what IP he was moved to: “moved to 69.55.239.XX”
- Any customer who’s the target of an attack will lose his attacked IP and be given a new IP from the “bad boy” block – 69.55.239.0/24. Any customer who’s machine was used to DoS attack someone else does not need to receive a new IP.
Incoming DOS:
- Send an “incoming dos” email to the customer to explain what happened. Take care to cc his alt contacts esp if his email looks like it’s hosted on our server. Optional- you may want to preemptively update his DNS so that any domains pointed at the old IP will point at the new badboy IP. The ipswap script is useful for this purpose if there are many domains.
- Switch out the attacked IP for the bad boy IP: for VPS’s, remove the old IP and add the new one. For dedicateds, tell them they will need to assign the new IP and remove the old one from their server. Offer an IPKVM if they’re nervous. Remind them that they may need to change the gateway IP if the attacked IP was the primary IP on their server.
- Once the attacked IP has been removed, respond to the DoS email from Castle telling them it’s ok to remove the null and what bad boy IP they were moved to (Castle needs confirmation on the move). Save the email from castle to the “dos” folder.
Outgoing DOS:
- Do a ps and top on the customer’s VPS to see what process is doing the DoS attack. It’s almost always obvious and usually includes an IP address as part of the process name. Stop his VPS. Send an “outgoing dos” email to the customer to explain what happened, showing them their ps, telling them we took them offline till we could (in the case of a jail) coordinate a time with them to bring it back online when we know they’ll be waiting to login and take action to patch/fix (in the case of a linux VPS remind them they can fire it up via control panel- give link…unless the hacker has root access in which case assume he can reach the cpl as well and disable it or block all traffic to all ports). It’s good in these cases to offer to block off all ssh access (in our firewall) except from an IP they give us until the compromise has been identified and removed, ESPECIALLY if the dos process is a real user (not apache, or some other service) cause the hacker could log back in the moment the server comes back up. Take care to cc his alt contacts esp if his email looks like it’s hosted on our server.
- Save the email to the “dos” folder.
- Work with the customer to definitively identify how the hacker got in in the first place (and to prevent a future occurrence). Possibly offer the customer a new VPS to move into since the old one is “tainted”. Encourage them to look at passwords and patch all software, esp web-based.
Notes on resizing gconcats (with growfs)
To figure out the new size of the a partition, subtract 16 from the c partition: 20G: 41943030 - 16 = 41943014 18G: 37748727 - 16 = 37748711 16G: 33554424 - 16 = 33554408 14G: 29360121 - 16 = 29360105 12G: 25165818 - 16 = 25165802 10G: 20971515 - 16 = 20971499 8G: 16777212 - 16 = 16777196 6G: 12582909 - 16 = 12582893 4G: 8388606 - 16 = 8388590 C partition: 20G: 4194304 * 10 - 10 = 41943030 18G: 4194304 * 9 - 9 = 37748727 16G: 4194304 * 8 - 8 = 33554424 14G: 4194304 * 7 - 7 = 29360121 12G: 4194304 * 6 - 6 = 25165818 10G: 4194304 * 5 - 5 = 20971515 8G: 4194304 * 4 - 4 = 16777212 6G: 4194304 * 3 - 3 = 12582909 4G: 4194304 * 2 - 2 = 8388606 or for 1G volumes: 2G: 4194302 - 16 = 4194286 2G: 2097152 * 2 - 2 = 4194302