firewall (newgateway)

Located at castle, this machine is the primary (only) firewall for the entire network at castle. It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).

The server is running FreeBSD 4.11 x86, has a 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card. Both drives are hotswap. Server has dual-power supplies. Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal).

Services Provided

firewall (ipfw)
snmp

Firewall Rule Configuration

See FreeBSD_Reference#Firewall_Rule_Configuration for more discussion on how to actually manipulate firewall rules.

Disaster Recovery

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"


#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248

Cronjobs

1 0 * * * /usr/local/etc/rsync.backup

Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3  4 5 17331

Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside /etc/daily.local you will see a call to /etc/makepiperules.pl This script will create /etc/firewall.sh which contains all the firewall and pipe rules in place at the time the script was run.

DOS attacks

See FreeBSD_Reference#Handling_a_DoS_attack regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will

enter a rule that allows all UDP to go through
send an emergency email to support and indicating an attack is in progress
send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1

This file lives under /usr/home/user/doswatch.pl To run:

cd /usr/home/user
./doswatch.pl &

To kill;

fg
^C

It writes its findings to /usr/home/user/doswatch.log

backup1

Located at castle, this machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server. It is running Ubuntu-Server 8.04 x86, and has a 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card. Its drives are hot-swap. Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)

Services provided

backup via rsync
mysql
nfs
snmp

Usage and Notes

all data is stored under /data
virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: /backup1), and they all have their ssh keys setup to allow passwordless rsync's
each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
cancelled customer systems are compressed and stored under /data/deprecated
archived bwdb2 flow files are stored under /data/bwdb2
critical files from backup2 are stored under /data/backup2

Cronjobs

00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate  /data/backuplog.log

this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup.

10,25,40,55 * * * * /usr/local/sbin/processsql.pl

this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.

0 0 * * * /usr/local/sbin/3wraidchk

checks the health of the RAID array

Regular maintenance

backup2

Located at castle, this machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. It is running FreeBSD 6.1 x86, and has the following arrays and controllers:

3ware 7500-8:

200 GB JBOD (1 x 200G) labeled 0-0
500 GB RAID5 (3 x 250G) 0-1 thru 0-3
700 GB RAID5 (4 x 250G) 0-4 thru 0-7

3ware 7500-8:

700 GB RAID5 (4 x 250G) 1-0 thru 1-3
700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit. All are hot-swap. Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC network at i2b and castle)

Services provided

backup via rsync and nfs
samba
nfs
snmp

Usage

all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: /mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4
iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /mnt/data2/iso
this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
this server serves as an archive for exported db data from bwdb and old flow files.
isys backs up here
customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
in /mnt/data4 there are lots of useful things used for building our vps servers, customer servers, and management scripts:
- /bin: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
- /build: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
- /vzrpms: contains the OS templates for many-to-most of the OS's we offer on vz systems

Cronjobs

backs itself up nightly to nfs-mounted backup1 (mountpoint: /backup2)

Regular maintenance

Check on health

backup3

Located at i2b, this machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups. It is running Ubuntu TODO, and has the following arrays and controllers:

All drives are hot-swap. Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2

Services provided

backup via rsync and nfs
samba
nfs
digi realport
snmp

Usage

all data is stored under /data
iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /data/iso
this server serves as an archive for exported db data from bwdb and old flow files.
inftrastructure machines at i2b back up here
customers are nfs-moutned under /data/customers as file-backed loopback devices

management scripts

mkbackups

Cronjobs

0 0 * * * /usr/local/sbin/3wraidchk

RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive

Rotate daily snapshots for infrastructure machine backups

Regular maintenance

Check on RAID health

Build

backup3 Configuration Last updated 2010-07-09

1. install Ubuntu 10.04 10G / ext3 2G swap ~ /data ext4

2. install packages apt-get update apt-get upgrade apt-get install gcc apt-get install libssl-dev apt-get install libncurses5-dev apt-get install cu apt-get install snmp snmpd ntp nfs-kernel-server

3. tweak grub, enable serial

vi /etc/default/grub

GRUB_HIDDEN_TIMEOUT=0

GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64" GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0" update-grub

echo "start on stopped rc RUNLEVEL=[2345] stop on runlevel [!2345]

respawn exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf

4. install realport driver

give the digi an ip with DgIpServ.exe

cd /usr/src/ wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz tar xzf dgrp-1.9.tgz cd dgrp-1.9/ ./configure make make install make postinstall update-rc.d dgrp_daemon defaults

5. configure ports dgrp_cfg_node init el 10.1.2.10 16

try with: cu -l /dev/ttyel00 -s 38400

6. shell, ntp, ssh key, hosts

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf server 10.1.2.1 server ntp.ubuntu.com

cd /root/ ssh-keygen -t dsa

echo "69.55.230.10 backup2" >> /etc/hosts echo "69.55.230.11 backup1" >> /etc/hosts echo "10.1.2.4 bwdb2" >> /etc/hosts echo "10.1.2.3 backup3" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

vi /root/.bashrc export PS1="[\u@\h \w]# " alias h='history' alias vi='vim' alias j='jobs' export PS1="[\u@\h \w]# " alias dr='screen -dr' export EDITOR=vim export GREP_OPTIONS='--color=auto' export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600' alias tip-firewall2='cu -l ttyel01 -s 115200' alias tip-nat2='cu -l /dev/ttyel02 -s 115200' alias tip-backup3='cu -l ttyel04 -s 38400' alias tip-bwdb2='cu -l ttyel03 -s 115200'

source /root/.bashrc

echo 'rocommunity jcread 10.1.4.5 rocommunity jcread 10.1.4.3 agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

6. nfs

echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

7. bb

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb chown bb.bb /home/bb

cd ~bb scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95 /var:90:95 /data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh (remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh

(y to questions)

./bbchkhosts.sh

(ignore ssh errors)

cd ../.. chown -R bb . su bb cd cd bbc1.9e-btf/src

make; make install

cd .. ./runbb.sh start more BBOUT (look for errors) exit

vi /etc/rc.local su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule: ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

vi ~/bbc1.9e-btf/etc/bbdef-client.sh DFWARN=199 DFPANIC=199

7. mysql

apt-get install mysql

mkdir /data/mysql chown mysql:mysql /data/mysql /etc/init.d/mysql stop mv /var/lib/mysql/* /data/mysql/ mv /data/mysql/ib_* /var/lib/mysql/ vi /etc/mysql/my.cnf (change datadir to /data/mysql) vi /etc/apparmor.d/usr.sbin.mysqld add:

 /data/mysql/ r,
 /data/mysql/** rwk,

Comment out:

/var/lib/mysql/ r,
/var/lib/mysql/** rwk,

/etc/init.d/apparmor restart /etc/init.d/mysql start

7. tw_cli and raid check

scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk vi /usr/local/sbin/checkraid.sh

%s/c0/c2/g

crontab -e 0 0 * * * /usr/local/sbin/3wraidchk

8. misc binaries scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive vi /usr/local/sbin/snapshot_archive (remove entries)

crontab -e 35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

9. mkbackup

mkdir /data/customers

cat > /usr/local/sbin/mkbackups

!/bin/sh

if test $1; then

 cid=$1

else

 echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
 exit

fi

if test $2; then

 gb=$2

else

 echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
 exit

fi

if test $3; then

 ip=$3

else

 echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
 exit

fi

if test -e /data/customers/${cid}-file; then

 echo "ERROR: /data/customers/${cid}-file exists"
 exit

else

 echo "touch /data/customers/${cid}-file"
 touch /data/customers/${cid}-file
 count=`echo $gb|awk '{print $1*1000}'`
 echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
 dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
 echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
 /sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file

fi

if test -e /data/customers/$cid; then

 echo "ERROR: /data/customers/$cid exists"
 exit

else

 echo "mkdir /data/customers/${cid}"
 mkdir /data/customers/${cid}
 echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
 mount -o loop /data/customers/${cid}-file /data/customers/$cid
 df -h /data/customers/$cid

 echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
 echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
 echo "" >> /etc/nfs_backup_mounts.sh

 echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
 /etc/init.d/nfs-kernel-server restart
 tail /var/log/messages

fi

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local add: /etc/nfs_backup_mounts.sh

10. samba

apt-get install samba

vi /etc/samba/smb.conf

comment out any mounts, add

[data]

  read only = yes
  locking = no
  path = /data/iso
  guest ok = yes

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

cd /data/iso scp backup2:/d2/iso/3wfirmware.iso . scp backup2:/d2/iso/MD5SUMS . scp backup2:/d2/iso/bootimg.iso . scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso . scp backup2:/d2/iso/win98bootcd.iso . scp backup2:/d2/iso/acronis_bootdisk.iso . scp backup2:/d2/iso/memtest86-3.2.iso .

11. add ns3c to resolv.conf and /etc/network/interfaces

Infrastructure Machines

Contents

firewall (newgateway)

Services Provided

Firewall Rule Configuration

Disaster Recovery

Cronjobs

DOS attacks

backup1

Services provided

Usage and Notes

Cronjobs

Regular maintenance

backup2

Services provided

Usage

Cronjobs

Regular maintenance

backup3

Services provided

Usage

management scripts

Cronjobs

Regular maintenance

Build

Navigation menu

Infrastructure Machines

firewall (newgateway)

Services Provided

Firewall Rule Configuration

Disaster Recovery

Cronjobs

DOS attacks

backup1

Services provided

Usage and Notes

Cronjobs

Regular maintenance

backup2

Services provided

Usage

Cronjobs

Regular maintenance

backup3

Services provided

Usage

management scripts

Cronjobs

Regular maintenance

Build

Navigation menu

Search