Infrastructure Machines

2012-11-21T23:22:34Z

99.133.163.50: /* Build */

= firewall (newgateway) =

Located at castle, this machine is the primary (only) firewall for the entire network at castle. It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).

The server is running FreeBSD 4.11 x86, has a 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card. Both drives are hotswap. Server has dual-power supplies. Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal).

== Services Provided ==
* firewall (ipfw)
* snmp

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

Located at castle, this machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server. It is running Ubuntu-Server 8.04 x86, and has a 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card. Its drives are hot-swap. Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup.
<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

= backup2 =
Located at castle, this machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. It is running FreeBSD 6.1 x86, and has the following arrays and controllers:

3ware 7500-8:
*200 GB JBOD (1 x 200G) labeled 0-0
*500 GB RAID5 (3 x 250G) 0-1 thru 0-3
*700 GB RAID5 (4 x 250G) 0-4 thru 0-7

3ware 7500-8:
*700 GB RAID5 (4 x 250G) 1-0 thru 1-3
*700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.
All are hot-swap. Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC network at i2b and castle)

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
Located at i2b, this machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups. It is running Ubuntu TODO, and has the following arrays and controllers:

All drives are hot-swap. Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/3wraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04
10G / ext3
2G swap
~ /data ext4</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'</pre>

Load new shell:
source /root/.bashrc

Setup snmpd:
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd</pre>

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== tw_cli and raid check ===

scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk

=== misc binaries ===
scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackups
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== add ns3c to resolv.conf and /etc/network/interfaces ===

JCWiki - User contributions [en]

Infrastructure Machines