Editing Infrastructure Machines (section)

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups. 

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply. 
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

mkbackup <cid> GB <ip>

== Cronjobs ==
 0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

 35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

 echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
        address 69.55.229.4
        netmask 255.255.255.0
        network 69.55.229.0
        broadcast 69.55.229.255
        gateway 69.55.229.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 69.55.229.3 66.181.0.2
        dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
        address 69.55.231.2
        netmask 255.255.255.0
        network 69.55.231.0
        broadcast 69.55.231.255

auto eth1
iface eth1 inet static
        address 10.1.2.3
        netmask 255.255.255.0
        network 10.1.2.0
        broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz 
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
 dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
 cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre> 

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
 source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity  jcread 10.1.4.5
rocommunity  jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
 echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
 /etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
 echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
 echo "bb:x:1984:" >> /etc/group
 pwconv

Create home:
 mkdir /home/bb
 chown bb.bb /home/bb
 cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
 echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
 echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

 vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

 chmod +r /var/log/messages

 ./bbchkcfg.sh 
(y to questions)
 ./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre> 
(look for errors)
 exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

 echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
 ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>


=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

 scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
 vi /usr/local/sbin/snapshot_archive
(remove entries)

 crontab -e
 35 4 * * * /usr/local/sbin/snapshot_archive

 scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
 scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];


my %mail = (
   To      => $to,
   From    => $to,
   Subject => $sub,
   Message => '',
   smtp    => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

 chmod 0700 /bin/mail

=== mkbackup ===

 mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
  cid=$1
else
  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
  exit
fi

if test $2; then
  gb=$2
else
  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
  exit
fi

if test $3; then
  ip=$3
else
  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
  exit
fi


if test -e /data/customers/${cid}-file; then
  echo "ERROR: /data/customers/${cid}-file exists"
  exit
else
  echo "touch /data/customers/${cid}-file"
  touch /data/customers/${cid}-file
  count=`echo $gb|awk '{print $1*1000}'`
  echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
  dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
  echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
  /sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
  echo "ERROR: /data/customers/$cid exists"
  exit
else
  echo "mkdir /data/customers/${cid}"
  mkdir /data/customers/${cid}
  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
  mount -o loop /data/customers/${cid}-file /data/customers/$cid
  df -h /data/customers/$cid

  echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
  echo "" >> /etc/nfs_backup_mounts.sh

  echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
  /etc/init.d/nfs-kernel-server restart
  tail /var/log/messages
fi</pre>

 chmod 0700 /usr/local/sbin/mkbackup

 vi /etc/rc.local
add:
 /etc/nfs_backup_mounts.sh

=== samba ===

 apt-get install samba

 vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
   read only = yes
   locking = no
   path = /data/iso
   guest ok = yes</pre>

 /etc/init.d/smbd restart

 mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
 rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
 for f in `find .  -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
 [root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
 [root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
 [root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs). 

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
 chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
 vi /usr/local/www/mgmt/mrtg/mrtg1.cfg