Editing NetHere (section)

=== Backup Server    ===

backup-1 -> odd numberd unix
backup-2 -> even numbered unix


On Saturdays, we do a level 0 dump
the rest are differential

Restoration fees are basically $25 per day restored for email, $150 for website from cancelled archive, $25 a day for website 

Restoration is based on customer request. In order to restore a site/mailbox fully, you must start restoration from the previous level 0 backup and continue until the day after the day requested, since backups are performed in the morning.

General rule is to leave the gzipped file available for 24 hours, after that remove it.


==== Site Restoral ====


If the site was removed by the web_del script, then so long as it is within 30 days since removal, the gzipped site will be available within /www/archive/www.domain.tld-date_removed.tar.gz
Extract with tar:
# tar -zxvpf www.domain.tld-date_removed.tar.gz

If the site needs to be restored from backup, here are the steps

1) To determine where backups are stored, go to /nethere/conf/backup

2) grep for the server within the directory to determine the backup directory used
i.e. 
# grep unixweb-1 *

4) Go to /backup1/unix/unixweb-1.nethere.net
  a) you'll see a bunch of directories, named <date>-<dump_level>

5) Since the backups happen in the morning, typically you'll need to go to the day before, restore
   e.g. for 11/10
  a) cd 20041109-3
  b) restore -if and whatever the filesystem is named
     (for interactive: add files/dirs, extract, then use 1 for volume #)
     i.e.
     restore> add <path_to_dir/files>
     restore> extract
     (for volume #: 1)
     (Set owner: y)

6) After you've restored the files, just use scp as root on sawfish to copy them off of backup-# and then onto unixweb-#
  a) (on sawfish - two step process)
     # scp backup-1:/tmp/<restored_file> /tmp
     # scp /tmp/<restored_files> unixweb-1:/tmp

7) Cleanup any restored files off of backup-# server and sawfish


With the newly restored files, following are the steps necessary to restore a site:

1) Restore www.domain.tld directory to /www
   # mv /www/archive/www.domain.tld /www

2) Restore configuration files
  a) Apache: mv /www/archive/nethere/conf/apache/(n)vhosts/www.domain.tld /nethere/conf/apache/conf/(n)vhosts/
  b) Webalizer: mv /www/archive/nethere/conf/webalizer/unix/www.domain.tld /nethere/conf/webalizer/unix

3) Add user to /etc/master.passwd:
  a) vipw
     (go to end of file, read in the master.passwd file from /www/www.domain.tld)
     :r /www/www.domain.tld/master.passwd
  b) Remove /www/www.domain.tld/master.passwd

4) Change flags on cgi-bin/ directories:
   # chflags sunlnk /www/www.domain.tld/htdocs/cgi-bin/
   # chflags schg /www/www.domain.tld/htdocs/cgi-bin/php*

5) Add configuration file entry to nvhosts/vhosts.conf
   # co -l (n)vhosts.conf
   # vi (n)vhosts.conf
   # ci -u (n)vhosts.conf

6) Check apache, restart; verify apache
   # apachectl configtest
   # apachectl graceful
   # 

7) Cleanup remaining restored files
   # rm -r /www/archive/www
   # rm -r /www/archive/nethere


8) Enter zone into DNS on ns1.nethere.net
  a) zone file
     # mv /named/archive/domain.tld /named/master
     # mv /named/archive/domain.tld,v /named/master/RCS
  b) named.master file 
     # co -l /named/named.master
     (add zone)
     # ci -u /named/named.master
     # cd /named; make new-zone
     # tail /var/log/named   (look for errors)
*************************************************
(12:40:47 PM) Henry Chan: restore is now available that is compatible with the 4.4bsd format
(12:41:04 PM) Henry Chan: to restore, use the following command: restore -c -i -f path_to_archive
(12:41:08 PM) Henry Chan: the "-c" is what does it
(12:41:29 PM) Henry Chan: (only applies to backup-1-new... doesn't work on backup-2 or backup-4)

 
==== Email Restoral ====
<pre>
Storage path prefixes:
	/nfs/1 is sndg-netapp-1 (on backup-4 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)
	/nfs/2 is sndg-netapp-2 (on backup-3 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)
	/nfs/3 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)
        /nfs/4 is sndg-netapp-1 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-1.nethere.net)
        /nfs/5 is sndg-netapp-2 (on backup-1 - /backup/hosts/2/e0.sndg-netapp-2.nethere.net)
        /nfs/6 is sndg-netapp-3 (on backup-2 - /backup/hosts/2/e0.sndg-netapp-3.nethere.net)

Looking up a mailbox's Storage Path:
	You will need to know which directory and NFS server the customer's mail is 
stored on. Use the Provisioning Tool to get the Storage Path (find the customer's mail
account, then click on the Engineering sub-tab under the Email tab). You should end up
with something like "/nfs/3/nh/h/t/htchan/Maildir". Match the prefix of the storage
directory with an NFS server (see prefixes above).



1) If restoring a recently deleted mail account:
	a) Look for the archive on mailbox-1 in /nfs/archive/mail/{platform}/{username}.{date}-{PID}.tar.gz. 
           If it doesn't exist, it's been too long and the only way to get email back is to restore from backup.
	b) Make sure the account is re-created in Provisioning Tool and look up the storage path 
           (see "Looking up a mailbox's Storage Path" above).
	c) Extract the archive to a temporary directory:
		Sample command:
			cd /tmp; tar xzvpf /nfs/archive/mail/nh/zella.20090902-7508.tar.gz
		Sample output:
			nfs/2/nh/z/e/zella/
			nfs/2/nh/z/e/zella/Maildir/
			nfs/2/nh/z/e/zella/Maildir/tmp/
			nfs/2/nh/z/e/zella/Maildir/new/
			nfs/2/nh/z/e/zella/Maildir/cur/
			nfs/2/nh/z/e/zella/Maildir/maildirsize
			nfs/2/nh/z/e/zella/Maildir/.Trash/
			nfs/2/nh/z/e/zella/Maildir/.Trash/tmp/
			nfs/2/nh/z/e/zella/Maildir/.Trash/new/
			nfs/2/nh/z/e/zella/Maildir/.Trash/cur/
			nfs/2/nh/z/e/zella/Maildir/.Trash/maildirfolder
			nfs/2/nh/z/e/zella/Maildir/.Drafts/
			nfs/2/nh/z/e/zella/Maildir/.Drafts/tmp/
			nfs/2/nh/z/e/zella/Maildir/.Drafts/new/
			nfs/2/nh/z/e/zella/Maildir/.Drafts/cur/
			nfs/2/nh/z/e/zella/Maildir/.Drafts/maildirfolder
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/tmp/
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/new/
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/cur/
			nfs/2/nh/z/e/zella/Maildir/.Sent Items/maildirfolder
			nfs/2/nh/z/e/zella/Maildir/courierpop3dsizelist
		Note:
			In the above output, notice that the Maildir is located in "nfs/2/nh/z/e/zella", it will be used in the next step.
	d) Copy the Maildir files to the new mailbox storage path using tar (tar does a better job at preserving 
           things like symbolic links and permissions than cp or mv):
		Sample command:
			cd /tmp/nfs/2/nh/z/e/zella; tar cf - Maildir | (cd /nfs/3/nh/z/e/zella; tar xvpf -)
		Output:
			You should list the files as it is copied. Basically, similar to what you 
                        saw in step "c" above but without the "nfs/2/nh/z/e/zella" prefix.
		Note:
			The command format is basically:
				cd /tmp/{restored-maildir-path-see-note-in-step-c}; tar cf - Maildir | (cd {new-storage-path-without-Maildir}; tar xvpf -)
	e) Clean up temporary directory:
		Sample command:
			cd /tmp; rm -rf nfs

2) If restoring a zfs mailbox (from the last week) to a particular date's backup:
        a) login to the correct sndg-netapp-[1/2/3]-new
        b) get the files from the correct zfs snapshot
           Sample commands:
             cd /tank0/mail/.zfs/snapshot/20180215-0/nh/c/h/christineat
             ls -l
           cd to the desired directory and copy the files over to the users mailbox
             cp -p /tank0/mail/.zfs/20180215-0/nh/c/h/christineat/Maildir/cur /tank0/mail/nh/c/h/christineat/Maildir/cur

3) If restoring a mailbox to a particular date's backup:
	a) Look up the storage path to the customer's mailbox and determine which backup server the 
           dump file is on (see "Looking up a mailbox's Storage Path" and "Storage path prefixes" above).
	b) Restore the level 0 and any incremental backups (in order) to /tmp on the backup server:
		Sample commands:
			cd /tmp
			restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090912-0/mail.dump
			chflags -R 0 /tmp
			restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090913-1/mail.dump
			chflags -R 0 /tmp
				...skipped repetitive stuff here...
			restore4x -if /backup/hosts/2/e0.sndg-netapp-2.nethere.net/20090918-6/mail.dump
			chflags -R 0 /tmp
		Note:
			After each restore, we need to recursively remove all flags from /tmp to eliminate 
                       the immutable flag that gets set on the files (this happens only on NetApp dumps... who knows why).
	c) Create a new tar of the Maildir directory so that it can be copied to mailbox-1 for further processing:
		Sample command:
			cd /tmp/nh/z/e/zella; tar cf /tmp/archive.tar Maildir
	d) Clean up /tmp:
		Sample command:
			rm -rf /tmp/nh
		Note:
			Depending on the platform, it might be /tmp/nh, /tmp/si, or /tmp/zn.
	e) Copy the /tmp/archive.tar file to /tmp on mailbox-1.
	f) On mailbox-1, extract the Maildir archive on top of what they already have:
		Sample command:
			cd /nfs/2/nh/z/e/zella; tar xvpf /tmp/archive.tar
	g) Remove /tmp/archive.tar.
</pre>

==== Beginning of old instructions ====
Use the same techniques as site restoration, with the following exceptions:

1) Since everything is in maildir format, you have to reassmble things in /tmp/<username>, tar it up, and then restore it on any of the mailbox machines
   a) using the following syntax: 
      # tar -zcpf /tmp/username.tgz yyyymmdd-#/platform yyyymmdd-#/platform yyyymmdd-#/platform
      i.e.
      # tar -zcpf /tmp/username.tgz 20050915-5/nh 20050914-4/nh 20050910-0/nh

2) In regards to mail spools, you'll need to use the chflags command to adjust the flags on the files. For whatever reason, they are stored with a system immutable flag
   a) Do chflags -R noschg  <dir> on the restored directory before copying the files with sawfish otherwise, the files cannot be deleted:
      i.e. 
      # chflags -R noschg /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/
   b) After tar/gzipping, remove the restored directories
      i.e
      # rm -r /backup1/unix/sndg-netapp-1-e2b.nethere.net/20050105-4/nh

3) Can restore quickly with script on mailbox-1 (or mailbox-2) using the NH script "restoremail"
   a) # /nethere/sbin/restoremail -h for usage
      i.e. 
      # restoremail -p nh -f nhusername.tgz -n 2 -u nhusername
*** End of old instructions ***



* Backup locations *

backup-1:
backup-3.nethere.net
fpweb-1.nethere.net
home-1.nethere.net
koi.nethere.net
marmaduke.inetworld.net
ntdb-1.nethere.net
ntweb-1.nethere.net
ntweb-3.nethere.net
ntweb-5.nethere.net
ntweb-7.nethere.net
phoenix.nethere.net
unixweb-1.nethere.net
unixweb-3.nethere.net
unixweb-5.nethere.net
unixweb-7.nethere.net
shark.nethere.net
tetra.nethere.net
wms-1.nethere.net

backup-2:
andromeda.nethere.net
backup-4.nethere.net
eel.nethere.net
fpweb-2.nethere.net
ntdb-2.nethere.net
ntweb-2.nethere.net
ntweb-4.nethere.net
ntweb-6.nethere.net
unixweb-2.nethere.net
unixweb-4.nethere.net
unixweb-6.nethere.net
ds.znet.com
mx1.znet.com
mx2.znet.com
mx3.znet.com
dmx.znet.com
la.znet.com
uf.znet.com

backup-3:
backup-1.nethere.net
lists-1.nethere.net
mailbox-1.mail.nethere.net
mailbox-3.mail.nethere.net
mta-1.mail.nethere.net
mx-1.nethere.net
nscache-1.nethere.net
nsrbl-1.nethere.net
pegasus.nethere.net
scan-1.mail.nethere.net
sndg-netapp-2.nethere.net
relay-1.mail.nethere.net
webmail-1.mail.nethere.net

backup-4:
ahi.nethere.net
backup-2.nethere.net
cp-1.nethere.net
cygnus.nethere.net
dragon.nethere.net
lisa.nethere.net
mailbox-2.mail.nethere.net
mta-2.mail.nethere.net
news-1.nethere.net
nscache-2.nethere.net
nsrbl-2.nethere.net
sawfish.nethere.net
scan-2.mail.nethere.net
scribe.nethere.net
sndg-netapp-1.nethere.net
relay-2.mail.nethere.net
urchin.nethere.net