FreeBSD Reference

Dump and Restore

Last updated 2005-04-12

One of the basic tasks that we need to perform in this organization is creating filesystem images and accurately cloning them across servers.

Since each server instance running on the machines has a full FreeBSD filesystem, it's important to understand the correct ways of cloning a filesystem and extracting the filesystem into multiple places.

The natural assumption is to use tar(1). However, tar(1) is not appropriate for archiving special files and for preserving all ownership and special attributes of certain files. This is important to keep in mind, because when we create a file system image, we recreate the entire OS. Run this command to see what type of "special" files are sometimes not treated correctly with tar(1):

`ls -asl /dev | more`

Please note the major/minor device numbers that these files have.

It would be difficult to describe all the shortcomings of tar(1), but they exist, and must be avoided by using different tools. An almost monthly question that appears on the FreeBSD mailing lists is "what should I use to safely backup this kind of filesystem or these kind of files and be sure that I am faithfully preserving everything." This is where dump(8) and restore(8) come in. A FreeBSD core member was once quoted on one of the mailing lists saying "Use dump or you will lose".

It is actually unfortunate that we cannot use tar(1) because it is much simpler, and more versatile. Further, many of the files that we are guaranteeing accuracy on by using dump(8) and restore(8) are probably not that important in the environment we run he server instances in. We provide a full /dev to our customers, but really they need very little of it. However, it's not for me to say why they are getting a FreeBSD server, and so I try to reproduce a real one as faithfully as possible.

---

Basically the dump(8) command dumps a filesystem to a regular file, and the restore(8) command restores the data from that file into a path you specify at some later date. dump(8) is non-versatile in the sense that you can only dump filesystems. So, you can dump /var (if you have /var as a seperate partition), but you cannot dump /var/db by itself, since presumably that is not on its own partition. If you want to dump /var/db, you need to dump all of /var. And if /var was not its own partition, you would have to dump all of / to get it. This is unfortunate, but it is how dump(8) works and there is no getting around it.

restore(8) is more flexible in the sense that you can restore a dump-file into any location. Just cd to where you want the dump-file expanded and expand it. Easy.

Let's give it a shot:

First, I run the command `df -k` to see what filesystems I have on my server:

www# df -k
Filesystem    1K-blocks     Used    Avail Capacity  Mounted on
/dev/mlxd0s1a   1016303   862925    72074    92%    /
/dev/mlxd0s1f   7030220  5488394   979409    85%    /mnt/data1
/dev/mlxd1s1e  17369623 11692971  4287083    73%    /mnt/data2
/dev/mlxd0s1e    508143   181167   286325    39%    /var
procfs                4        4        0   100%    /proc
www#

I'll choose the /var partition for this example since it is small and will go quickly. The syntax for dumping a filesystem to a file is:

`dump -0a -f /mnt/data2/dump-file /dev/mlxd0s1e`

So, we are dumping to a file /mnt/data2/dump-file that does not already exist, and we are dumping from the device /dev/mlxd0s1e which, as you can see from the df(1) output is the device that /var is mounted on. It is worth repeating that you can only dump an entire filesystem, and that filesystem is referred to by the device it is mounted on.

www# dump -0a -f /mnt/data2/dump-file /dev/mlxd0s1e
DUMP: Date of this level 0 dump: Tue Aug 20 00:21:25 2002
DUMP: Date of last level 0 dump: the epoch
DUMP: Dumping /dev/mlxd0s1e (/var) to /mnt/data2/dump-file
DUMP: mapping (Pass I) [regular files]
DUMP: mapping (Pass II) [directories]
DUMP: estimated 181700 tape blocks.
DUMP: dumping (Pass III) [directories]
DUMP: dumping (Pass IV) [regular files]
DUMP: DUMP: 181684 tape blocks on 1 volume
DUMP: finished in 52 seconds, throughput 3493 KBytes/sec
DUMP: Closing /mnt/data2/dump-file
DUMP: DUMP IS DONE
www#

So now we have a backup of /var in a dump-file named /mntdata2/dump-file. We can just save that until we need it - it is ok to compress it or tar it up or even rename it - it is simply a regular file.

Now we wish to restore the contents of dump-file using the restore(8) command. Remember that we can restore the file anywhere – we can restore it on a partition like we dumped from (such as /var) or we can restore it in a deeply nested directory such as /usr/local/etc/rc.d. It will restore correctly regardless of where you restore it.

www#
www# mkdir /mnt/data2/test
www# mkdir /mnt/data2/test/test
www# cd /mnt/data2/test/test
www# restore -x -f /mnt/data2/dump-file

You have not read any tapes yet. Unless you know which volume your file(s) are on you should start with the last   volume and work towards the first.
Specify next volume #: 1
set owner/mode for '.'? [yn] y
www#

These are probably the only arguments you will ever use for restore(8) unless you place it in a script which we will discuss in a bit. You'll note that two pieces of user input are needed in this process, as opposed to dump(8) which did not ask any questions. First, we are asked to "Specify next volume #:", to which we reply "1" (and we will almost certainly always simply reply with "1", since we are not doing multi volume tape sets or anything complicated lke that). Second, many seconds later after the dump is complete, we are asked if we want to "set owner/mode for '.'?" and the answer is Yes, or more specifically, the y key followed by a carriage return. The whole point of using these tools is to preserve attributes, so it is natural that we would say yes to this question.

Now that you’ve learned how to do it manually, you’ll be happy to know there’s a script that automatically enters the 1 and y for you: dumprestore <path-to-dump-file>

however…this sometimes fails to set proper permissions. So either followup or don’t use it.

---

Some additional information that you should know:

First, dump(8) and restore(8) are not just FreeBSD commands. They are very old, historical unix commands. They have the ability to interact with complex systems of tapes/volumes and are the core of a lot of enterprise backup schemes. You would be surprised how many unix admins use these tools instead of big fancy backup products.

Second, whereas the only syntax we will probably ever use with dump(8) is this:

dump -0a -f </target/file> </device/name>

there is another syntax that we will sometimes use with restore(8). In our example, we restored in an interactive environment - that is, we are logged onto a server and running restore(8) in a shell. We use this syntax:

restore -x -f /some/dump-file

and we then answer the two questions that require an interactive response. However, sometimes we will want to run restore(8) out of a script, which presents a problem because we cannot answer the questions in a script when it runs. Therefore, if we restore a dump-file as part of a shell script, we use this syntax instead:

`restore -rf /some/dump-file`

This is just slightly different - it performs the restore without needing any interaction. However, it also leaves behind a file called `restoresymtable` in the directory where you restore the dump-file. This file can be safely deleted.

One other consideration that is worth noting is that dump(8) does not save mount points of other partitions. If you dump / and /var is a separate partition, when you restore the dump-file, you will not have a /var directory in the directory where you restore the dump-file.

---

So now you should try these things out on your own. Install FreeBSD on a system and dump the entire / filesystem into a file. Strange as it may seem, you can actually dump / into /. For instance, if /data is not its own partition and is simply a directory under /, you can dump / into /data/dump-file.

After dumping / into a dump-file, dump /var into a file called dump-file-var. Now create any old directory and restore the dump-file. Then create a /var directory inside the target where you restored your dump-file, cd to it, and restore the dump-file-var file.

Presuming you had no other meaningful partitions besides /var that were on their own device, you now have a clone of your entire system inside /data (or wherever you restored things).

ipfw (firewalling)

ipfw is a userland command (/sbin/ipfw) that is, in a sense, the FreeBSD equivalent of the linux `iptables` command. Although the ipfw command is present in all FreeBSD installations, it also requires kernel support.

You can enable ipfw by adding this line to your kernel configuration line:

options IPFIREWALL

and building and installing that new kernel. It should be noted, however, that there are two other options that should probably be added to your kernel configuration file as well:

options IPFIREWALL_VERBOSE
options IPFIREWALL_VERBOSE_LIMIT=100

The first allows ipfw to do verbose logging of packets and events, and the second limits the number of log entries that a single, consecutive event can log.

ipfw support can also be loaded as a module, by simply running:

/sbin/kldload ipfw.ko

It is VERY IMPORTANT to understand that ipfw, when existent in a kernel, or when loaded as a kernel module with kldload, will _always_ have a single default, final rule. That is, there will always be at least one rule present, and that rule will be the final, catch-all rule for any packet that does not match any prior rules.

By default, the final, default rules is:

deny ip from any to any

Which means (VERY IMPORTANT) that if you enable the firewall in your kernel and then reboot, you will have no network connectivity to the machine in question _at all_. The machine will have a single rule, which is set to deny all ip traffic, and you will not be able to reach it in any way.

Similarly, if you simply run `/sbin/kldload ipfw.ko` on a normal FreeBSD system, you will be instantly locked out of the system, and the system will be completely unreachable from the network.

There are a few ways to solve this problem. First, you can add a fourth line to your kernel configuration file:

options IPFIREWALL_DEFAULT_TO_ACCEPT

This means that ipfw support will be added to the kernel, but with a final, default rule of:

allow ip from any to any

With this rule in place as the final, default rule, even if you add no other rules, when the system restarts you will be able to see it on the network just fine, and no packets will be blocked.

Another way to solve the problem is to leave ipfw in the kernel set to the default (which is to deny ip from any to any) but a rule to your startup configuration to open up some or all network services. The default will still be to deny ip from any to any, but any traffic the first matches your allow rules will be allowed in. This method will work fine, regardless of whether you added ipfw support in the kernel or by loading a module. In fact, you can (theoretically, although I distrust this method) run this command:

/sbin/kldload ipfw.ko ; ipfw add 65500 allow ip from any to any

and not be locked out ... the default rule is always number 65535, so by instantly adding a rule at 65500 to allow all traffic, you can load the ipfw module safely without locking yourself out. I do not recommend this, however, as I do not trust that this will work 100% of the time.

Finally, a third method to enable ipfw without locking yourself out is to rebuild the ipfw.ko module to set its default to allow, instead of deny - just like we were able to set a kernel configuration line to set the default to allow. Building this module with that custom configuration is beyond the scope of this document.

Why does FreeBSD have a default setting for ipfw to deny all traffic ? The reason is, you do not want to allow a malicious party to circumvent your firewall rules by crashing your machine. In some configurations, firewall rules may not be loaded at boot time, so if the default was "allow all from any to any" then one could circumvent the firewall rules by crashing the firewall - when it came back up the rules would not be loaded, and all traffic would pass.

REMEMBER - no matter what you do, there will always be a single rule in place - rule number 65535. You cannot delete this rule, and it is set to deny all or allow all depending on how it was set (as we discussed above).

You can view all of the rules currently active on a system by running:

/sbin/ipfw show

A useful way to see all the rules that might apply to a certain IP, for instance 10.10.10.10, is:

/sbin/ipfw show | grep "10.10.10.10"

Here is what the results of `ipfw show` would look like on a system with a default accept configuration:

# ipfw show
65535 109490 44385056 allow ip from any to any

As you can see, there is only one rule - number 65535. You can only configure 65535 rules, and that is always the last one, and is always present.

The second number in the output is the packet count - how many packets have passed through that rule, and the third number is the byte count - how many bytes have passed through that rule.

You add rules with commands like:

ipfw add 100 allow tcp from any to 10.10.10.10 22

That line allows all tcp traffic on port 22 destined for 10.10.10.10.

If you now run ipfw show, you would see:

# ipfw show
00100      0        0 allow tcp from any to 10.10.10.10 22
65535 116103 46464923 allow ip from any to any

You could then delete rule #100 with:

ipfw del 00100

A very useful and command setup for a system behind a firewall is to open up only the ports that correspond to services actually running on that system, and deny all other traffic. Here is an example, where the IP in question is 10.10.10.10:

ipfw add 100 allow tcp from any to 10.10.10.10 established
ipfw add 200 allow tcp from any to 10.10.10.10 22,25,80,443 setup
ipfw add 300 deny tcp from any to 10.10.10.10

So, in this example, we first allow any previously established tcp connections to this IP, then we allow any tcp connections that are in a setup (TCP 3-way handshake) mode - these two rules together account for all possible legitimate tcp traffic. Then the third rule simply denies all other tcp traffic.

As you can see, ipfw applies rules FIFO - so as a packet travels from rule 0 to rule 65535, as soon as it matches a rule, the packet is processed, and leaves the ruleset - no further rules are applied to that packet, as it has been passed.

In the above example, the user is running ssh, smtp, http, and https ... however, for a system to be workable on the network, a few other things should be open if they run a dns server - since dns uses UDP as well:

ipfw add 100 allow tcp from any to 10.10.10.10 established
ipfw add 200 allow tcp from any to 10.10.10.10 22,25,53,80,443 setup
ipfw add 300 allow udp from any to 10.10.10.10 53
ipfw add 400 deny ip from any to 10.10.10.10

Note the addition of rule 300, which allows udp to come in on port 53. Also notice that since dns uses both tcp and udp, we have added 53 to the allowed tcp list as well. Finally, note that we can refer to ipfw rule numbers as either 00200 or 200 - it is the same thing.

Also, note that the final rule for this IP, #400, does not simply deny all tcp or all udp (or both), rather, it denies all IP _period_. So if it doesn't match the tcp list, and if it is not udp53, then it gets dropped as soon as it hits rule 400.

It should be noted that there is performance penalty on the firewall machine for each rule added to it _that packets pass through_. (so, even if you have 1000 rules on the firewall, if the first rule is `allow ip from any to any` then there is no difference in performance than if the ruleset had only one rule - all packets get passed right away and none pass trough the other 999 rules)

If a poor ruleset design is in use, and the firewall takes in a lot of traffic (or passes a DoS attack) it can knock the firewall off the network. The machine will not be crashed - as soon as the traffic lets up it will respond again - but no communication will pass while the CPU is overloaded.

The number one consideration when tuning the ruleset for performance is to pass packets that should be passed as fast as possible, and dump packets that should be dumped as fast as possible. For example, we know that we don't want to ever pass certain types of packets (a tcp packet with all option fields set, a tcp packet with no MSS setting, and so on) so, the very first four rules on our firewall are:

00003   49913883 2009604713 deny tcp from any to any tcpflags syn tcpoptions !mss
00003   23958169 1342587681 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
00003        142       8496 deny tcp from any to any tcpflags syn,fin
00003          0          0 deny tcp from any to any tcpflags fin,psh,rst,urg

This means that these packets, if we ever encounter them (and we frequently do in a DoS attack) are dropped immediately. If we had 1000 rules in our ruleset, and put these at the end, a big DoS attack would cripple the firewall, because _every single_ other rule would have to be processed against every single packet until it got spit out the end.

On the other side of the coin, we also allow what we know should be allowed _as fast as we can_.

Take the example above:

ipfw add 100 allow tcp from any to 10.10.10.10 established
ipfw add 200 allow tcp from any to 10.10.10.10 22,25,80,443 setup
ipfw add 300 deny tcp from any to 10.10.10.10

Now, lets say we have four machines on our network, so we set up a block of rules for all four:

ipfw add 100 allow tcp from any to 10.10.10.10 established
ipfw add 150 allow tcp from any to 10.10.10.10 22,25,80,443 setup
ipfw add 200 deny tcp from any to 10.10.10.10

ipfw add 250 allow tcp from any to 10.10.10.20 established
ipfw add 300 allow tcp from any to 10.10.10.20 22,25,80,443 setup
ipfw add 350 deny tcp from any to 10.10.10.20

ipfw add 400 allow tcp from any to 10.10.10.30 established
ipfw add 450 allow tcp from any to 10.10.10.30 22,25,80,443 setup
ipfw add 500 deny tcp from any to 10.10.10.30

ipfw add 550 allow tcp from any to 10.10.10.40 established
ipfw add 600 allow tcp from any to 10.10.10.40 22,25,80,443 setup
ipfw add 650 deny tcp from any to 10.10.10.40

Now, each ruleset applies to a particular IP. The problem is, we can tell by looking at this that we _always_ pass established tcp packets – no matter what IP they are for. If an established packet comes in for 10.10.10.40, it first has to pass through 9 other rules before it gets to:

ipfw add 550 allow tcp from any to 10.10.10.40 established

Since we know we always pass established packets, we can shorten this dramatically by doing this:

ipfw add 001 allow tcp from any to any established

ipfw add 150 allow tcp from any to 10.10.10.10 22,25,80,443 setup
ipfw add 200 deny tcp from any to 10.10.10.10

ipfw add 300 allow tcp from any to 10.10.10.20 22,25,80,443 setup
ipfw add 350 deny tcp from any to 10.10.10.20

ipfw add 450 allow tcp from any to 10.10.10.30 22,25,80,443 setup
ipfw add 500 deny tcp from any to 10.10.10.30

ipfw add 600 allow tcp from any to 10.10.10.40 22,25,80,443 setup
ipfw add 650 deny tcp from any to 10.10.10.40

Now an established tcp connection (that we always want to pass) is passed at rule #1 - as fast as it possible can be passed. If we have hundreds of systems behind this firewall, with rulesets such as this, this can cause the CPU of the firewall to be totally idle, vs. being 50% utilized – just by putting that change in. I have seen it. This is true because a large percentage of all traffic at all times is established tcp traffic.

Also, note that we always deny ip from any to (the IP) as soon as that IPs ruleset is over - this is not technically necessary - you could just let the final 65535 "deny all" rule catch it and deny it there - but that would kill performance because the packet would have to be matched against every other rule inbetween first.

If you forget to put in index #, it will add as 65535

Notes on resizing gconcats (with growfs)

To figure out the new size of the a partition, subtract 16 from the c partition:
20G: 41943030 - 16 = 41943014
18G: 37748727 - 16 = 37748711
16G: 33554424 - 16 = 33554408
14G: 29360121 - 16 = 29360105
12G: 25165818 - 16 = 25165802
10G: 20971515 - 16 = 20971499
8G: 16777212 - 16 = 16777196
6G: 12582909 - 16 = 12582893
4G: 8388606 - 16 = 8388590
C partition: 20G: 4194304 * 10 - 10 = 41943030
18G: 4194304 * 9 - 9 = 37748727
16G: 4194304 * 8 - 8 = 33554424
14G: 4194304 * 7 - 7 = 29360121
12G: 4194304 * 6 - 6 = 25165818
10G: 4194304 * 5 - 5 = 20971515
8G: 4194304 * 4 - 4 = 16777212
6G: 4194304 * 3 - 3 = 12582909
4G: 4194304 * 2 - 2 = 8388606
or for 1G volumes: 2G: 4194302 - 16 = 4194286 2G: 2097152 * 2 - 2 = 4194302