Editing Customer Backups

___TOC___

Backups are performed for customers upon request. When someone does ask, they should really only be asking for real user data dirs to be backed up, though we won’t refuse any reasonable request. 

Our backup program does not follow symlinks so, for instance, if someone asks for backups on FreeBSD, and one of the directories they want backed up is /home, remember that that is just a symlink to /usr/home - so when you enter it into the backup file on their jail you need to enter it as /usr/home,

Backup entries are stored in
<tt>/usr/local/jail/bin/backup.config</tt> (FreeBSD)
<tt>/usr/local/sbin/backup.config</tt> (linux)

The first line in the file refers to where the backups go on the backup server, the second line is the backup server they go to. Each line beyond that is an actual directory to be  backed up (vb is an alias to vi the backup file):

<pre>jail15# vb

#!/bin/sh
backupdir=/data/jail15
server=backup1

## ENTRY /mnt/backup
## ENTRY /etc
## ENTRY /usr/local/etc
## ENTRY /usr/local/jail
## ENTRY /root/logs
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/etc
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/root
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/usr/home</pre>

When adding new directories, make sure the dirs backing up really exist by ls’ing them

To search for existing backups:

<pre>jail15# gb col00214
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/etc
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/root
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/usr/home
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/usr/local
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/var/log
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/www
## ENTRY /mnt/data1/69.55.238.98-col00214-DIR/var/spool/postfix</pre>

The backup script runs nightly at various times, but generally between midnight at 3am. If the backup script tries to backup a dir that doesn’t exist, an email will be  sent to the support queue.

The mechanism behind backups is thus:

When the backups are to be run, a script called <tt>backup</tt> is run. It does 2 things:
* update all our scripts from the repository on backup2
* start the actual backup script

The backup script reads the config file and rsync’s directories defined against a corresponding directory on the backup server specified. So, if the first line in the config file is <tt>backupdir=/data/jail15</tt>Then backups are made to <tt>backup1:/data/jail15/0</tt> 
Because these backups are incremental and destructive, only changed files are updated and files which don’t exist locally are removed from backup1. This makes the backups very fast and efficient. 

After backups are made from customer machines to the backup server, a housekeeping script fires off on the backup server which deletes the backups made 7 nights ago, rotates (renames) nights -1 thru -6 “down” in the count, and makes a snapshot of the previous night’s backup. Specifically, here’s what the script does:
          <pre>rm -fr [dir]/6
          mv [dir]/5 [dir]/6
          mv [dir]/4 [dir]/5
          mv [dir]/3 [dir]/4
          mv [dir]/2 [dir]/3
          mv [dir]/1 [dir]/2
          gcp -al [dir]/0 [dir]/1</pre>

This means at any point during the day, there are 6-7 days worth of backups for any given machine.

= Restoring backups =

The backups are copied to their respecitve directories on the backup servers nightly, and the houseclean script continuously rotates (effectively deleting all backup files older than 2 days) so it s very important to respond to backup restore requests '''immediately''' - if they sit for too long, they could get automatically deleted.

We do not replace the backup files on their system - we simply make and copy the tarball that they require onto their system and put it in their / directory.  When you respond to their email telling them that their backup has been copied onto their system, always tell them that they should place the tarball in a temp directory and explode it there, so they do not overwrite other things already on their system.

Also, it is a good idea to make a temp directory on the backup server and place the last few days of that persons backups in it, so they do not get deleted by the housecleaning script - this is helpful if the person deleted the tarball accidently, or something else goes wrong.