Editing Infrastructure Machines (section)

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server. 

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply. 
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql - traffic data
* nfs server - for backups
* snmp client - for big brother
* bigbrother client

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate  /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb 
12G boot
4664 GB 

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number  Start  End  Size  File system  Name  Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number  Start   End     Size    File system  Name     Flags
# 1      17.4kB  4987GB  4987GB               primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
#        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
#        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
#        102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables:   967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem            Size  Used Avail Use% Mounted on
#/dev/sda2             8.3G  540M  7.3G   7% /
#varrun               1013M   40K 1013M   1% /var/run
#varlock              1013M     0 1013M   0% /var/lock
#udev                 1013M   56K 1013M   1% /dev
#devshm               1013M     0 1013M   0% /dev/shm
#/dev/sdb1             4.5T  192M  4.3T   1% /data


apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3        backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty 
# 
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0 


vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines: 
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity  jcread 10.1.4.5
rocommunity  jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface


echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab


vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh 
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT 
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"


echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart


echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd


echo '10.1.4.8                backup1' >> /etc/hosts
echo '/dev/sdb1	/data  ext3  rw,noatime  0  0' >> /etc/fstab


to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server 
apt-get install make
apt-get install gcc
apt-get install g++	
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
  /data/mysql/ r,
  /data/mysql/** rwk,
Comment out:
#  /var/lib/mysql/ r,
#  /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on


0 0 * * * /usr/local/sbin/3wraidchk
</pre>