Infrastructure Machines: Difference between revisions
Jump to navigation
Jump to search
| Line 2,633: | Line 2,633: | ||
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl | 2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl | ||
Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database: | |||
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl | |||
== Regular maintenance == | == Regular maintenance == | ||
Revision as of 15:55, 24 January 2013
Summary
This machine is the swiss army knife of the company, playing host to many services and functions.
- Location: castle, cab 3-8
- OS: FreeBSD 4.10 x86
- Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
- Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
- Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
Services Provided
- web
- mysql
- bigbrother server/pager
- snmp
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains Other addresses (aliases) are defined per /etc/mail/aliases
The following active users have mail hosted on this server:
- dave
- linux
- support
- payments
- sales
- tech1
- info
Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.
Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
# more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:
:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo
:0
! -t
}
control: cd /etc/mail; make stop (stop), cd /etc/mail; make start (start)
The following aliases are also in place:
debian: linux jobs: info careers: info #reboot: 6128102202@txt.att.net #reboot: 8582298897@vtext.com reboot: pager #pager: 8582298897@vtext.com pager: 4158718324@txt.att.net tech1on: "| /usr/local/sbin/tech1on.sh" tech1off: "| /usr/local/sbin/tech1off.sh"
To change them, edit /etc/aliases and then run newaliases
Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.
web
See Management System / Public Website / Signup
mysql
mysql 4.1.22 is running on port 3306
- datadir: /mnt/data1/db/mysql/
- config: /etc/my.cnf
- database: jc
- control: /usr/local/etc/rc.d/mysql-server.sh stop (stop), /usr/local/etc/rc.d/mysql-server.sh start (start)
bigbrother
There is a client running on mail (which monitors the services running on mail and mail itself), installed under /usr/home/bb/bbc1.9e-btf
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under /usr/home/bb/bbsrc/bb1.9i-btf
Both are running under the user bb
Refer to BigBrother for more about use.
DNS (ns1c.johncompanies.com)
ns1c is a jail running on the mail server, who's IP is 69.55.225.225
It's running from /mnt/data1/ns1c-dir
See DNS for more details
Usage and Notes
- always mounted to backup1 and backup2 via nfs:
backup2:/mnt/data1 on /backup (nfs) backup2:/mnt/data2 on /backup2 (nfs) backup2:/mnt/data3 on /backup3 (nfs) backup2:/mnt/data4 on /backup4 (nfs) backup1:/data on /backup1 (nfs)
Cronjobs
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs
*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs
40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs
41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs
1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last 1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last 2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1 15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1 1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth
1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script
0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats
15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png 16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM
10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report
*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues
5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date
7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)
8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1
16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)
*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only
25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers
10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch
10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow
0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay
0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail
0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys
Regular maintenance
nat
Summary
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.
- Location: castle, cab 3-7
- OS: FreeBSD 4.10 x86
- Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
- Hardware: Custom 1U. single power supply.
- Drives: one 8 GB IDE drive
Services Provided
- nat
nat control
All rules are contained in and look like:
cat /etc/ipnat.rules # www (was 69.55.230.12) # virt19 #bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32 # virt18 #bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32 # virt13 #bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32 # virt12 #bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32 # virt17 bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32 # virt11 #bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32 # ASA #bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32 # P1A bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32 #bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32 # developer (was 69.55.230.17) # jail2 #bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32 # jail8 #bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32 # jail9 #bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32 # POLL #BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32 # 1U SUN #BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32 # ?? #BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32 # random machine #bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32 #bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32 # OFFICE OUTBOUND TRAFFIC #map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp #map fxp0 10.1.6.0/24 -> 0.0.0.0/32
A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197
To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules
You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.
nat2
Summary
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.
- Location: i2b, cab 6
- OS: FreeBSD 6.4 x86
- Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
- Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
- Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.
Services Provided
- nat
- bigbrother
- ns3c (jail)
nat config
Here's what's currently nat'd on nat2:
cat /etc/ipnat.rules # sample entry #ATS-9 bimap em0 10.1.2.79/32 -> 66.181.18.14/32 #ATS-8 bimap em0 10.1.2.78/32 -> 66.181.18.13/32 #ATS-7 bimap em0 10.1.2.77/32 -> 66.181.18.12/32 #ATS-6 bimap em0 10.1.2.76/32 -> 66.181.18.6/32 #ATS-5 bimap em0 10.1.2.75/32 -> 66.181.18.7/32 #ATS-4 bimap em0 10.1.2.74/32 -> 66.181.18.8/32 #ATS-3 bimap em0 10.1.2.73/32 -> 66.181.18.9/32 #ATS-2 bimap em0 10.1.2.72/32 -> 66.181.18.10/32 #ATS-1 bimap em0 10.1.2.71/32 -> 66.181.18.11/32 #bwdb2 bimap em0 10.1.2.4/32 -> 66.181.18.5/32 # spare map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp #bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32 #bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
build
- partition map:
/ 512m swap 1G /var 256m /tmp 256m /usr 5g /mnt/data1 ~
- edit /etc/make.conf
echo "WITHOUT_X11=yes \ KERNCONF=nat2 \ BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
- add settings to /boot/loader.conf and /boot.config
echo "-Dh" >> /boot.config echo 'console="comconsole,vidconsole" \ boot_multicons="YES" \ boot_serial="YES" \ comconsole_speed="115200"' >> /boot/loader.conf
- turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys ttyv2 "/usr/libexec/getty Pc" cons25 off secure ttyv3 "/usr/libexec/getty Pc" cons25 off secure ttyv4 "/usr/libexec/getty Pc" cons25 off secure ttyv5 "/usr/libexec/getty Pc" cons25 off secure ttyv6 "/usr/libexec/getty Pc" cons25 off secure ttyv7 "/usr/libexec/getty Pc" cons25 off secure # Serial terminals # The 'dialup' keyword identifies dialin lines to login, fingerd etc. ttyd0 "/usr/libexec/getty std.9600" vt100 on secure kill -1 1
on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into) test serial console
- populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts echo "69.55.230.11 backup1" >> /etc/hosts echo "10.1.2.4 bwdb2" >> /etc/hosts echo "10.1.2.3 backup3" >> /etc/hosts
- put key in authorized_keys on backup3
cd ssh-keygen -t dsa -b 1024
(default location, leave password blank)
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup3 and backup 2 without getting a login prompt
ssh backup3 hostname
ssh backup2 hostname
ssh backup1 hostname
- edit root's path and login script:
vi /root/.cshrc
Change alias entries (add G):
alias la ls -aG alias lf ls -FAG alias ll ls -lAG alias ls ls -AG alias mbm mb mount alias mbu mb umount
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
- install cvsup
cd /usr/ports/net/cvsup-without-gui make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
- get latest sources for this release:
cd /usr/src echo "*default host=cvsup4.freebsd.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_6_4\ *default delete use-rel-suffix\ *default compress\ src-all" > sup cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
- configure new kernel.
cd /usr/src/sys/i386/conf scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2
- build, install kernel and world
cd /boot mv kernel kernel.GENERIC cd kernel.GENERIC cd /usr/src make buildkernel installkernel make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null make installworld mergemaster -i
- populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf hostname="nat2.johncompanies.com" kern_securelevel_enable="NO" portmap_enable="NO" sendmail_enable="NO" usbd_enable="YES" xntpd_enable="YES" nfs_client_enable="YES" nfs_reserved_port_only="YES" ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0" #ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0" #ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255" ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0" ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255" ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0" defaultrouter="10.1.6.1" #defaultrouter=" 66.181.14.250" snmpd_enable="YES" ipnat_enable="YES" ipnat_rules="/etc/ipnat.rules" gateway_enable="YES" inetd_enable="YES" inetd_flags="-wW -a 10.1.2.1" fsck_y_enable="YES" background_fsck="NO" sshd_enable="YES"
- reboot. Confirm new kernel is loaded
uname -a
- update ports:
cd /usr/ports echo "*default host=cvsup4.FreeBSD.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_6_4\ *default delete use-rel-suffix\ *default compress\ ports-all tag=." > sup cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
- Install raid mgmt tool
cd /usr/local/sbin fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz tar xzf tw_cli-freebsd-x86-9.5.3.tgz rm tw_cli-freebsd-x86-9.5.3.tgz chmod 0700 tw_cli
Test:
./tw_cli info c0
- install rsync from ports
cd /usr/ports/net/rsync make install clean
choose default options
- install perl from ports
cd /usr/ports/lang/perl5.8 make install clean
- install screen from ports
cd /usr/ports/sysutils/screen make install clean
- install bb client
adduser Username: bb Full name: bb Uid (Leave empty for default): 1984 Login group [bb]: Login group is bb. Invite bb into other groups? []: Login class [default]: Shell (sh csh tcsh nologin) [sh]: Home directory [/home/bb]: Use password-based authentication? [yes]: Use an empty password? (yes/no) [no]: Use a random password? (yes/no) [no]: yes Lock out the account after creation? [no]: Username : bb Password : <random> Full Name : bb Uid : 1984 Class : Groups : bb Home : /home/bb Shell : /bin/sh Locked : no OK? (yes/no): yes cd /usr/home/bb scp backup2:/mnt/data4/build/bb/bb-freebsd.tar . tar xvf bb-freebsd.tar
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM
cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$
./runbb.sh start
more BBOUT
(look for errors)
exit
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):
ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2
- configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts 64.163.14.48 nat2.johncompanies.com # ssh su bb cd bbsrc/bb/runbb.sh restart ; exit
- configure ntp
echo "server 69.55.230.2 server 0.pool.ntp.org server 1.pool.ntp.org server 2.pool.ntp.org server 3.pool.ntp.org server 66.187.233.4 server 217.204.76.170 server 64.112.189.11 server 66.69.112.130 server 80.85.129.25 server 80.237.234.15 server 130.60.7.44 server 134.99.176.3 server 198.144.202.250 server 202.74.170.194 server 204.17.42.199 server 204.87.183.6 server 213.15.3.1 server 213.239.178.33 server 217.114.97.97 server 69.55.230.2" > /etc/ntp.conf
/usr/sbin/ntpd -A -p /var/run/ntpd.pid sleep 2; ntpq -p
(confirm it’s able to reach our time server)
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh chmod 0700 /usr/local/etc/rc.d/ntp.sh
- fwd and reverse lookups on ns1c
vr johncompanies.com (edit the PTR too)
- setup backups, nfs mount
mkdir /backup3 echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab echo '#\!/bin/sh\ backupdir=/data/nat2/current\ \ ## ENTRY /etc ' > /usr/local/etc/backup.config
on backup3: setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive
scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup vi /usr/local/etc/rsync.backup backup1 > backup3
crontab -e 1 0 * * * /usr/local/etc/rsync.backup
- edit sshd_config for security
vi /etc/ssh/sshd_config ListenAddress 66.181.18.1 ListenAddress 69.55.229.2 ListenAddress 10.1.2.1 kill -1 `cat /var/run/sshd.pid`
- raid chk
cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl
my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;
foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}
- netflow stuff
add crontab entries
crontab -e 30 3 * * * /usr/local/etc/rsync.backup 0 0 1 * * /sbin/ipfw zero 0 0 1 * * /sbin/ipfw del 3 4 5 59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count 3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count 0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; */5 * * * * /usr/local/sbin/lsiraidchk #10 0 * * * rm /var/spool/clientmqueue/*
scp /etc/makefwrules.pl user@64.163.14.48:~ scp /etc/makepiperules.pl user@64.163.14.48:~ mv /home/user/makefwrules.pl /etc mv /home/user/makepiperules.pl /etc touch /etc/firewall.sh mkdir /etc/oldrules/
other binaries
scp /usr/local/bin/rulemaker user@64.163.14.48:~ mv ~user/rulemaker /usr/local/sbin scp ~user/Sendmail.pm user@64.163.14.48:~ scp ~user/doswatch.pl user@64.163.14.48:~
- add nat rules
vi /etc/ipnat.rules # sample entry bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32 #bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32 ipnat -C -f /etc/ipnat.rules
- shell for user
cp /root/.cshrc ~user/ vi ~user/
change # to $
- mrtg
cd /usr/ports/net-mgmt/mrtg make install clean
(no FONTCONFIG, v3)
this didn't work cause of libtool incompat
so manually moved files:
scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/ scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/ scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/ scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/ scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/ scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2 scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2 rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/ scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4 rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net cd /usr/ports/net/rrdtool make install mkdir -p /mnt/data1/mrtg/data scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/ scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/ cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg cat > /mnt/data1/mrtg/mrtg.sh #!/bin/sh /usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg chmod 0700 /mnt/data1/mrtg/mrtg.sh crontab -e * * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null
- snmp firewall block
cat > /usr/local/etc/rc.d/boot.sh ipfw add 10 allow udp from 69.55.230.2 to any 161 ipfw add 10 allow udp from 10.1.2.1 to any 161 ipfw add 11 deny udp from any to any 161 chmod 0700 /usr/local/etc/rc.d/boot.sh
bwdb
Summary
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.
- Location: castle, cab 3-7
- OS: FreeBSD 4.10 x86
- Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
- Hardware: Custom 1U. Single power supply.
- Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.
Services Provided
- netflow
- mysql
- bigbrother
- snmp
netflow
The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under /usr/home/flows/ (organized by date). The flow file contains all traffic data for a 15min increment of time.
A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl
into a processing queue: /usr/home/working
Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl
mysql
The database storing all the traffic data is named traffic Tables:
mysql> show tables; +---------------------------+ | Tables_in_traffic | +---------------------------+ | dailyIpTotals_69_55_224 | | dailyIpTotals_69_55_225 | | dailyIpTotals_69_55_226 | | dailyIpTotals_69_55_227 | | dailyIpTotals_69_55_228 | | dailyIpTotals_69_55_229 | | dailyIpTotals_69_55_230 | | dailyIpTotals_69_55_231 | | dailyIpTotals_69_55_232 | | dailyIpTotals_69_55_233 | | dailyIpTotals_69_55_234 | | dailyIpTotals_69_55_235 | | dailyIpTotals_69_55_236 | | dailyIpTotals_69_55_237 | | dailyIpTotals_69_55_238 | | dailyIpTotals_69_55_239 | | dailyPortTotals_69_55_224 | | dailyPortTotals_69_55_225 | | dailyPortTotals_69_55_226 | | dailyPortTotals_69_55_227 | | dailyPortTotals_69_55_228 | | dailyPortTotals_69_55_229 | | dailyPortTotals_69_55_230 | | dailyPortTotals_69_55_231 | | dailyPortTotals_69_55_232 | | dailyPortTotals_69_55_233 | | dailyPortTotals_69_55_234 | | dailyPortTotals_69_55_235 | | dailyPortTotals_69_55_236 | | dailyPortTotals_69_55_237 | | dailyPortTotals_69_55_238 | | dailyPortTotals_69_55_239 | | ipTotals_69_55_224 | | ipTotals_69_55_225 | | ipTotals_69_55_226 | | ipTotals_69_55_227 | | ipTotals_69_55_228 | | ipTotals_69_55_229 | | ipTotals_69_55_230 | | ipTotals_69_55_231 | | ipTotals_69_55_232 | | ipTotals_69_55_233 | | ipTotals_69_55_234 | | ipTotals_69_55_235 | | ipTotals_69_55_236 | | ipTotals_69_55_237 | | ipTotals_69_55_238 | | ipTotals_69_55_239 | | portTotals_69_55_224 | | portTotals_69_55_225 | | portTotals_69_55_226 | | portTotals_69_55_227 | | portTotals_69_55_228 | | portTotals_69_55_229 | | portTotals_69_55_230 | | portTotals_69_55_231 | | portTotals_69_55_232 | | portTotals_69_55_233 | | portTotals_69_55_234 | | portTotals_69_55_235 | | portTotals_69_55_236 | | portTotals_69_55_237 | | portTotals_69_55_238 | | portTotals_69_55_239 | +---------------------------+
So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.
mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
The id is a unique identifier (key), direction indicates incoming or outgoing traffic (outbound = 2, inbound = 1), octets are the amount of traffic in kilobytes, and packets is the total number of packets.
The 15-minute table has similar information, but it's organized in 15 minute increments:
mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+
mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4
So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.
That covers the simple traffic tabulation tables. We also track traffic by port:
mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)
mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30
mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3
This is largely the same with 2 more additions: protocol (1=ICMP, 6=TCP, 17=UDP), and port which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.
Regular maintenance
- Check RAID array
- archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.
archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.
- if space becomes tight, move flow files and exported data to a backup server, both located in /usr/home/flowbin/archive and /usr/home/exported, respectively
Build
BIOS Config
disable quiet boot
set to last state after power loss
set date/time to GMT
enable serial console output (baud rate 115200)
Install OS
Install FreeBSD 8.3 amd64
- partition map:
/ 500m swap 2048m /var 256m /tmp 256m /usr ~
- edit /etc/make.conf
Castle:
echo "WITHOUT_X11=yes \ KERNCONF=bwdb \ BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
i2b:
echo "WITHOUT_X11=yes \ KERNCONF=bwdb2 \ BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
- add settings to /boot/loader.conf and /boot.config
echo "-Dh" >> /boot.config echo 'console="comconsole,vidconsole" \ boot_multicons="YES" \ boot_serial="YES" \ comconsole_speed="115200"' >> /boot/loader.conf
- turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
vi /etc/ttys ttyv2 "/usr/libexec/getty Pc" cons25 off secure ttyv3 "/usr/libexec/getty Pc" cons25 off secure ttyv4 "/usr/libexec/getty Pc" cons25 off secure ttyv5 "/usr/libexec/getty Pc" cons25 off secure ttyv6 "/usr/libexec/getty Pc" cons25 off secure ttyv7 "/usr/libexec/getty Pc" cons25 off secure # Serial terminals # The 'dialup' keyword identifies dialin lines to login, fingerd etc. ttyu0 "/usr/libexec/getty std.9600" vt100 on secure kill -1 1
on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into) test serial console
- populate hosts
i2b:
echo "69.55.230.10 backup2" >> /etc/hosts echo "69.55.230.11 backup1" >> /etc/hosts echo "10.1.2.3 backup3" >> /etc/hosts
castle:
echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
- put key in authorized_keys on backup1 and backup2
cd ssh-keygen -t dsa -b 1024
(default location, leave password blank)
castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt
ssh backup1 hostname ssh backup2 hostname
- edit root's path and login script:
vi /root/.cshrc
Change alias entries (add G):
alias la ls -aG alias lf ls -FAG alias ll ls -lAG alias ls ls -AG
and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "
- install cvsup
cd /usr/ports/net/cvsup-without-gui make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null
- get latest sources for this release:
cd /usr/src echo "*default host=cvsup4.freebsd.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_8_3\ *default delete use-rel-suffix\ *default compress\ src-all" > sup cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null
- configure new kernel
cd /usr/src/sys/amd64/conf scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb
Edit config and change name:
vi bwdb ident bwdb
- build, install kernel and world
cd /boot mv kernel kernel.GENERIC cd kernel.GENERIC cd /usr/src make buildkernel installkernel make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null (2450: 1:56min, supermicro: 59mins, 2950: 38mins) make installworld (2450: 3min, supermicro: 1min, 2950: :34) mergemaster -i
- populate /etc/rc.conf with IPs and NFS settings
castle:
vi /etc/rc.conf hostname="bwdb.johncompanies.com" kern_securelevel_enable="NO" portmap_enable="NO" sendmail_enable="NO" usbd_enable="YES" xntpd_enable="YES" nfs_client_enable="YES" nfs_reserved_port_only="YES" ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0" ifconfig_em0="up promisc" defaultrouter="10.1.4.1" snmpd_enable="YES" inetd_enable="YES" inetd_flags="-wW -a 10.1.4.203" fsck_y_enable="YES" background_fsck="NO" sshd_enable="YES" ipfw_load="YES"
i2b:
vi /etc/rc.conf hostname="bwdb2.johncompanies.com" kern_securelevel_enable="NO" portmap_enable="NO" sendmail_enable="NO" usbd_enable="YES" xntpd_enable="YES" nfs_client_enable="YES" nfs_reserved_port_only="YES" ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0" ifconfig_em0="up promisc" defaultrouter="10.1.2.1" snmpd_enable="YES" inetd_enable="YES" inetd_flags="-wW -a 10.1.2.4" fsck_y_enable="YES" background_fsck="NO" sshd_enable="YES" ipfw_load="YES"
- reboot. Confirm new kernel is loaded
uname -a
- update ports:
cd /usr/ports echo "*default host=cvsup4.FreeBSD.org\ *default base=/usr\ *default prefix=/usr\ *default release=cvs tag=RELENG_8_3\ *default delete use-rel-suffix\ *default compress\ ports-all tag=." > sup cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null
- Install raid mgmt tool
cd /usr/local/sbin scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz . tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz rm tw_cli-freebsd-x86_64-9.5.0.1.tgz chmod 0700 tw_cli
Test:
./tw_cli info c0
Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc
Setup cronjob:
crontab -e */5 * * * * /usr/local/sbin/3wraidchk
- install rsync from ports
cd /usr/ports/net/rsync make install clean
choose default options
- install perl from ports
cd /usr/ports/lang/perl5.8 make install clean
choose default options
- install bb client
Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/
adduser
Output/response:
Username: bb Full name: bb Uid (Leave empty for default): 1984 Login group [bb]: Login group is bb. Invite bb into other groups? []: Login class [default]: Shell (sh csh tcsh nologin) [sh]: Home directory [/home/bb]: Use password-based authentication? [yes]: Use an empty password? (yes/no) [no]: Use a random password? (yes/no) [no]: yes Lock out the account after creation? [no]: Username : bb Password : <random> Full Name : bb Uid : 1984 Class : Groups : bb Home : /home/bb Shell : /bin/sh Locked : no OK? (yes/no): yes
cd /usr/home/bb scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz . tar xzf bb-freebsd_linuxcompat.tgz
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \ 10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
Edit for machine name and private IP.
if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \ 10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM
Edit for machine name.
cd /usr/home/bb/bbc1.9e-btf/etc ./bbchkcfg.sh (y to questions) ./bbchkhosts.sh (ignore ssh errors) cd ../.. chown -R bb . su bb cd cd bbc1.9e-btf ./runbb.sh start more BBOUT (look for errors) exit
Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh chmod +x /usr/local/etc/rc.d/bb.sh
If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:
ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2
- configure bb on mail
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts 10.1.4.203 bwdb.johncompanies.com # ssh su bb cd bbsrc/bb/runbb.sh restart ; exit
- configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf
I2b:
echo "server 10.1.2.1" > /etc/ntp.conf
/usr/sbin/ntpd -p /var/run/ntpd.pid sleep 2; ntpq -p
(confirm it’s able to reach our time server)
echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh chmod 0700 /usr/local/etc/rc.d/ntp.sh
- fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)
- setup backups
echo '#\!/bin/sh\ backupdir=/data/bwdb/current\ \
- ENTRY /etc ' > /usr/local/etc/backup.config
Castle: setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_archive
I2b: setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive
Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/
Edit rsync.backup and change config var to point to correct config file location: /usr/local/etc/backup.config
crontab -e 5 0 * * * /usr/local/etc/rsync.backup
- make /root/logs
mkdir /root/logs
- edit sshd_config for security
vi /etc/ssh/sshd_config ListenAddress 10.1.4.203 PermitRootLogin yes kill -1 `cat /var/run/sshd.pid`
Edit for private IP.
netflow
Install flow tools:
cd /usr/ports/net-mgmt/flow-tools make install clean
Defaults.
mkdir /usr/home/flows
Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh
chmod 0700 /usr/local/etc/rc.d/flow-capture.sh
Edit for private IP.
Netgraph start script:
cat > /usr/local/etc/rc.d/netgraph.sh /usr/sbin/ngctl -f- <<-SEQ mkpeer em0: netflow lower iface0 name em0:lower netflow connect em0: netflow: upper out0 mkpeer netflow: ksocket export inet/dgram/udp msg netflow:export connect inet/10.1.4.203:4444 SEQ #/usr/sbin/ngctl -f- <<-SEQ #shutdown netflow: #SEQ chmod 0700 /usr/local/etc/rc.d/netgraph.sh
Edit for private IP.
Confirm netflow is running after running scripts:
newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl Available commands: config get or set configuration of node at <path> connect Connects hook <peerhook> of the node at <relpath> to <hook> debug Get/set debugging verbosity level dot Produce a GraphViz (.dot) of the entire netgraph. help Show command summary or get more help on a specific command list Show information about all nodes mkpeer Create and connect a new node to the node at "path" msg Send a netgraph control message to the node at "path" name Assign name <name> to the node at <path> read Read and execute commands from a file rmhook Disconnect hook "hook" of the node at "path" show Show information about the node at <path> shutdown Shutdown the node at <path> status Get human readable status information from the node at <path> types Show information about all installed node types write Send a data packet down the hook named by "hook". quit Exit program + show netflow: Name: netflow Type: netflow ID: 00000004 Num hooks: 3 Local hook Peer name Peer type Peer ID Peer hook ---------- --------- --------- ------- --------- export <unnamed> ksocket 00000005 inet/dgram/udp out0 em0 ether 00000001 upper iface0 em0 ether 00000001 lower +
process flow tools
mkdir /usr/home/flowbin mkdir /usr/home/working
Install modules:
cd /usr/ports/devel/p5-Date-Calc make install clean cd /usr/ports/mail/p5-Mail-Sendmail make install clean
Queue script:
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl
use strict;
BEGIN {
push @INC, "/usr/home/flowbin";
}
use date;
my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";
my ($date, $time) = date::CurrentDateTime();
my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;
if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}
sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
Date.pm module:
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc
package date;
use strict;
use Date::Calc qw(:all);
sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}
sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}
sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}
sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}
sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}
sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}
sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}
sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}
sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}
sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}
sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);
# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}
sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);
my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}
sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}
sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}
sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}
sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}
sub ValiDate {
return @_;
}
sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}
1; #don't remove this line
chmod 0700 /usr/home/flowbin/queue.pl
Setup cronjob:
crontab -e #move flow data into the queue 1,16,31,46 * * * * /usr/home/flowbin/queue.pl
flow processing: i2b
cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl
#use strict;
#$debug=1;
#$dry=1;
my $log = '/usr/home/flowbin/discards.log';
use Data::Dumper;
BEGIN {
push @INC, "/usr/home/flowbin";
}
#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";
unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}
opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);
foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};
&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;
#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1
my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];
if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }
if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}
my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }
my $classC = "$ip[0]_$ip[1]_$ip[2]";
# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }
&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}
# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;
sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226
# all port/daily totals before 2005-04-07
This script sends the sql files to the traffic server for processing:
cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl
#use strict;
#$debug=1;
#$dry=1;
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my $remote = "backup1";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}
opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);
foreach my $file (sort @files) {
next unless $file =~ /bz2$/;
my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}
`rm $sqldir/.lock` unless $dry;
if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}
sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;
# prepare message
my %mail = (
To => 'support@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}
sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226
# all port/daily totals before 2005-04-07
crontab -e #process flows 2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl #move sql commands to traffic db 8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
flow processing: castle
cat > /usr/home/flowbin/processflows.pl
#!/usr/bin/perl
#use strict;
#$debug=1;
#$dry=1;
my $log = '/usr/home/flowbin/discards.log';
use Data::Dumper;
BEGIN {
push @INC, "/usr/home/flowbin";
}
use db;
#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}
my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};
opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);
foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};
&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;
#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1
my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];
if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }
if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}
my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }
my $classC = "$ip[0]_$ip[1]_$ip[2]";
# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;
$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}
sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }
&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;
$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}
# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;
sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226
# all port/daily totals before 2005-04-07
cat > /usr/home/flowbin/db.pm
#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;
use strict;
use DBI;
sub new {
my $class = shift;
my $self = {};
$self->{'debug'} = 0;
bless $self, $class;
return $self;
}
sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;
my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}
eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}
sub query {
my $self = shift;
my $query = shift;
$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}
sub disconnect {
my $self = shift;
$self->{'dbh'}->disconnect;
return 0;
}
sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;
if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
mkdir /usr/home/archive mkdir -p /usr/home/sql/tmp
crontab -e #process flows 2,17,32,47 * * * * /usr/home/flowbin/processflows.pl
setup traffic db
- Install mysql:
cd /usr/ports/databases/mysql50-server make install clean
cat >> /etc/rc.conf mysql_enable="YES"
Move db data dir:
/usr/local/etc/rc.d/mysql-server stop mkdir /usr/home/database/ mv /var/db/mysql/* /usr/home/database/ chown -R mysql:mysql /usr/home/database
Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}
/usr/local/etc/rc.d/mysql-server start
- Install mysql perl database modules:
cd /usr/ports/databases/p5-DBI make install clean cd /usr/ports/databases/p5-DBD-mysql50 make install clean (no to SSL support)
- Setting up database
rehash /usr/local/etc/rc.d/mysql-server start mysql -u root create database traffic; grant all on *.* to root@localhost identified by '5over3'; grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';
If this was a new server we'd setup new tables. See mysql for how those tables would be setup.
We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/
When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.
process flows from bwdb2
On traffic database server (bwdb):
mkdir -p /usr/home/bwdb2/pending/
crontab -e #import sql from bwdb2 10,25,40,55 * * * * /usr/local/sbin/processsql.pl
Add access to mysql:
mysql -u root -p grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
cat > /usr/local/sbin/processsql.pl
#!/usr/bin/perl
#use strict;
#$debug=1;
#$dry=1;
my $sqldir = "/data/bwdb2/pending";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}
opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);
foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}
`rm $sqldir/.lock` unless $dry;
if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}
sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;
# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}
sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
chmod 0700 /usr/local/sbin/processsql.pl
bwdb2
Summary
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.
- Location: i2b, cab6
- OS: FreeBSD 6.4 x86
- Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
- Hardware: Custom 2U. Single power supply.
- Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.
Services Provided
- netflow
- bigbrother
netflow
The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under /usr/home/flows/ (organized by date). The flow file contains all traffic data for a 15min increment of time.
A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl
into a processing queue: /usr/home/working
Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
Regular maintenance
- if space becomes tight, move sql files and flow files to backup server, both located in /usr/home/flowbin/archive
firewall (newgateway)
Summary
This machine is the primary (only) firewall for the entire network at castle.
- Location: castle, cab 3-8
- OS: FreeBSD 4.11 x86
- Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
- Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
- Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.
Services Provided
- firewall (ipfw)
- snmp
- bigbrother
Firewall Rule Configuration
See Firewall Rule Configuration for more discussion on how to actually manipulate firewall rules.
Disaster Recovery
If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.
Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)
hostname="newgateway.johncompanies.com" firewall_script="/etc/firewall.sh" firewall_enable="NO" sendmail_enable="NONE" sshd_enable="YES" inetd_enable="NO" xntpd_enable="YES" snmpd_enable="YES" #snmpd_flags="-as -p /var/run/snmpd.pid" #ipnat_enable="YES" #ipnat_rules="/etc/ipnat.rules" gateway_enable="YES" defaultrouter="69.55.233.161" ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0" ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248" # # Original JohnCompanies 69.55.224.0/20 # ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248" static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1 2 route13 route14 route15 route16 route17 route18" route_route1="-net 69.55.224.0 69.55.233.153" route_route2="-net 69.55.225.0 69.55.233.153" route_route3="-net 69.55.226.0 69.55.233.153" route_route4="-net 69.55.227.0 69.55.233.153" route_route5="-net 69.55.228.0 69.55.233.153" route_route6="-net 69.55.229.0 69.55.233.153" route_route7="-net 69.55.230.0 69.55.233.153" route_route8="-net 69.55.231.0 69.55.233.153" route_route9="-net 69.55.232.0 69.55.233.153" route_route10="-net 69.55.233.0 69.55.233.153" route_route11="-net 69.55.234.0 69.55.233.153" route_route12="-net 69.55.235.0 69.55.233.153" route_route13="-net 69.55.236.0 69.55.233.153" route_route14="-net 69.55.237.0 69.55.233.153" route_route15="-net 69.55.238.0 69.55.233.153" route_route16="-net 69.55.239.0 69.55.233.153" route_route17="-net 10.1.5.0 10.1.4.2" route_route18="-net 10.1.6.0 10.1.4.2" #In case of 3750 failure: #defaultrouter="69.43.128.81" #ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248" #bind .1's here: #ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0" #ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0" #ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0" #ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0" #ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0" #ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0" #ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0" #ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0" #ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0" #ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0" #ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0" #ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0" #ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0" #ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0" #ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0" #ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0" #bulk: # reassign 69.55.231.1 to the int iface on the firewall # set the DG on the firewall to 69.43.138.9 # set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248
Cronjobs
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1
0 0 1 * * /sbin/ipfw zero 0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).
Inside /etc/daily.local you will see a call to /etc/makepiperules.pl This script will create /etc/firewall.sh which contains all the firewall and pipe rules in place at the time the script was run.
DOS attacks
See Handling a DoS attack regarding how to handle a DOS attack.
Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
- enter a rule that allows all UDP to go through
- send an emergency email to support and indicating an attack is in progress
- send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
- wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl To run:
cd /usr/home/user ./doswatch.pl &
To kill;
fg ^C
It writes its findings to /usr/home/user/doswatch.log
backup1
Summary
This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.
- Location: castle, cab 3-8
- OS: Ubuntu 8.04.1 server x86
- Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
- Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
- Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card
Services provided
- backup via rsync
- mysql
- nfs
- snmp
- bigbrother
Usage and Notes
- all data is stored under /data
- virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: /backup1), and they all have their ssh keys setup to allow passwordless rsync's
- each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
- at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
- cancelled customer systems are compressed and stored under /data/deprecated
- archived bwdb2 flow files are stored under /data/bwdb2
- critical files from backup2 are stored under /data/backup2
Cronjobs
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null 35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup.
10,25,40,55 * * * * /usr/local/sbin/processsql.pl
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
0 0 * * * /usr/local/sbin/3wraidchk
checks the health of the RAID array
Regular maintenance
backup2
Summary
This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.
- Location: castle, cab 3-7
- OS: FreeBSD 6.1 x86
- Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
- Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
- Drives:
- 3ware 7500-8:
- 200 GB JBOD (1 x 200G) labeled 0-0
- 500 GB RAID5 (3 x 250G) 0-1 thru 0-3
- 700 GB RAID5 (4 x 250G) 0-4 thru 0-7
- 3ware 7500-8:
- 700 GB RAID5 (4 x 250G) 1-0 thru 1-3
- 700 GB RAID5 (4 x 250G) 1-4 thru 1-7
- 3ware 7500-8:
All drives MUST be western digital IDE drives. Other brands will not fit.
Services provided
- backup via rsync and nfs
- samba
- nfs
- snmp
- bigbrother
Usage
- all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: /mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4
- iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /mnt/data2/iso
- this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
- this server serves as an archive for exported db data from bwdb and old flow files.
- isys backs up here
- customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
- in /mnt/data4 there are lots of useful things used for building our vps servers, customer servers, and management scripts:
- /bin: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
- /build: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
- /vzrpms: contains the OS templates for many-to-most of the OS's we offer on vz systems
Cronjobs
- backs itself up nightly to nfs-mounted backup1 (mountpoint: /backup2)
Regular maintenance
backup3
Summary
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.
- Location: i2b, cab 6
- OS: Ubuntu 10.04.1 server amd64
- Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
- Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
- Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port
Services provided
- backup via rsync and nfs
- samba
- nfs
- digi realport
- snmp
- bigbrother
Usage
- all data is stored under /data
- iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /data/iso
- this server serves as an archive for exported db data from bwdb and old flow files.
- inftrastructure machines at i2b back up here
- customers are nfs-moutned under /data/customers as file-backed loopback devices
management scripts
- mkbackups
Cronjobs
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks
35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups
Regular maintenance
Build
BIOS Config
disable quiet boot
set to last state after power loss
set date/time to GMT
enable serial console output (baud rate 115200)
Install OS
Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat) 10G / ext3 2G swap ~ /data ext4 Install packages: openssh samba
DNS and private IP
echo "nameserver 69.55.225.225" >> /etc/resolv.conf
Add a 2nd IP to eth0 and setup priv net
vi /etc/network/interfaces
auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com
auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255
auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255
Install packages
apt-get update apt-get upgrade apt-get install gcc apt-get install libssl-dev apt-get install libncurses5-dev apt-get install cu apt-get install unzip apt-get install snmp snmpd ntp nfs-kernel-server
tweak grub, enable serial
vi /etc/default/grub #GRUB_HIDDEN_TIMEOUT=0 GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64" GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0" update-grub
echo "start on stopped rc RUNLEVEL=[2345] stop on runlevel [!2345] respawn exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf
install realport (digi) driver
give the digi an ip with DgIpServ.exe
cd /usr/src/ wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz tar xzf dgrp-1.9.tgz cd dgrp-1.9/ ./configure make make install make postinstall update-rc.d dgrp_daemon defaults
configure ports:
dgrp_cfg_node init el 10.1.2.10 16
try connecting with:
cu -l /dev/ttyel00 -s 38400
shell, ntp, ssh key, hosts
Shell autocompletion search:
echo "\"\e[5~\": history-search-backward" >> ~/.inputrc echo "\"\e[6~\": history-search-forward" >> ~/.inputrc
Setup ntp:
vi /etc/ntp.conf server 10.1.2.1 server ntp.ubuntu.com
Generate ssh keys:
cd /root/ ssh-keygen -t dsa
Defaults, no password
Setup hosts:
echo "69.55.230.10 backup2" >> /etc/hosts echo "69.55.230.11 backup1" >> /etc/hosts echo "10.1.2.4 bwdb2" >> /etc/hosts echo "10.1.2.3 backup3" >> /etc/hosts
Copy keys to servers where we need passwordless login:
cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
Setup shell:
vi /root/.bashrc (add to bottom) alias h='history' alias vi='vim' alias j='jobs' export PS1="[\u@\h \w]# " alias dr='screen -dr' export EDITOR=vim export GREP_OPTIONS='--color=auto' export HISTFILESIZE=1000 alias tip-switch-p20='cu -l ttyel00 -s 9600' alias tip-switch-p21='cu -l ttyel15 -s 9600' alias tip-switch-p22='cu -l ttyel14 -s 9600' alias tip-switch-p23='cu -l ttyel05 -s 9600' alias tip-switch-p24='cu -l ttyel06 -s 9600' alias tip-switch-p25='cu -l ttyel09 -s 9600' alias tip-switch-p26='cu -l ttyel07 -s 9600' alias tip-switch-p27='cu -l ttyel08 -s 9600' alias tip-firewall2='cu -l ttyel01 -s 115200' alias tip-nat2='cu -l /dev/ttyel02 -s 115200' alias tip-backup3='cu -l ttyel04 -s 38400' alias tip-bwdb2='cu -l ttyel03 -s 115200' alias tip-backup4='cu -l ttyel13 -s 115200' alias tip-jail3='cu -l ttyel11 -s 115200' Load new shell: source /root/.bashrc Setup snmpd (this is only valid for a server at castle): echo 'rocommunity jcread 10.1.4.5 rocommunity jcread 10.1.4.3 agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf to see which iface it is, on backup2: snmpwalk -v 1 -c jcread 10.1.4.8 interface === nfs === Allow mounts from private net: echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports Restart nfsd: /etc/init.d/nfs-kernel-server restart === bb === Add user, group: echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd echo "bb:x:1984:" >> /etc/group pwconv Create home: mkdir /home/bb chown bb.bb /home/bb cd ~bb Copy over and install files: <pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar . tar xf bb-linux.tar cd /home/bb/bbc1.9e-btf/etc
Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts
Configure low disk alerts:
echo "/:90:95 /var:90:95 /data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab
vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)
chmod +r /var/log/messages
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../.. chown -R bb . su bb cd cd bbc1.9e-btf/src
make; make install cd .. ./runbb.sh start more BBOUT
(look for errors)
exit
vi /etc/rc.local su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
(before the exit 0)
echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd
Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984
vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh DFWARN=199 DFPANIC=199
raid check
3ware
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk vi /usr/local/sbin/checkraid.sh :%s/c0/c2/g crontab -e 0 0 * * * /usr/local/sbin/3wraidchk
areca
cd /tmp wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip unzip linuxcli_V1.10.0_120815.zip cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/ chmod 0700 /usr/local/sbin/cli64 cli64 rsf info
scp backup2:/data4/bin/arecaraidchk /usr/local/sbin scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin crontab -e 0 0 * * * /usr/local/sbin/arecaraidchk
cat > /root/verify.sh cli64 vsf info cli64 rsf info cli64 disk info cli64 event info echo press enter when ready to run verify ; read x cli64 vsf check vol=1
misc binaries
scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive vi /usr/local/sbin/snapshot_archive
(remove entries)
crontab -e 35 4 * * * /usr/local/sbin/snapshot_archive
scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone
Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:
cat > /bin/mail #!/usr/bin/perl use strict; use warnings; use lib '/usr/local/sbin'; use Sendmail qw(sendmail); my $sub = $ARGV[1]; my $to = $ARGV[2]; my %mail = ( To => $to, From => $to, Subject => $sub, Message => '', smtp => 'mail.johncompanies.com' ); sendmail(%mail) || print "Error: $Sendmail::error";
chmod 0700 /bin/mail
mkbackup
mkdir /data/customers
cat > /usr/local/sbin/mkbackup
#!/bin/sh
if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi
if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi
if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi
if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi
if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid
echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh
echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi
chmod 0700 /usr/local/sbin/mkbackup
vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh
samba
apt-get install samba
vi /etc/samba/smb.conf
- comment out any mounts, add
[data] read only = yes locking = no path = /data/iso guest ok = yes
/etc/init.d/smbd restart
mkdir /data/iso
Bring over some stuff from backup2
cd /data/iso scp backup2:/d2/iso/3wfirmware.iso . scp backup2:/d2/iso/MD5SUMS . scp backup2:/d2/iso/bootimg.iso . scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso . scp backup2:/d2/iso/win98bootcd.iso . scp backup2:/d2/iso/acronis_bootdisk.iso . scp backup2:/d2/iso/memtest86-3.2.iso .
Moving from one server to another
Here are the steps you would take to move settings and data from one server to a new backup server:
- rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done
- copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh
- copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local
- copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports
- edit /etc/hostname on both machines (set current to oldbackup3)
- edit /etc/network/interfaces (swap IPs).
- stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh
- reboot both servers @ same time
- check everything out
- run /etc/nfs_backup_mounts.sh on new server
- if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
console
Summary
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.
- Location: castle, cab 3-8
- OS: SunOS 5.8 (solaris)
- Networking: Priv IP: 10.1.4.4
- Hardware: Sun Netra
To connect to consoles, ssh in as user 'console' and use the tip command to connect to devices listed in /etc/remote
i.e.
tip switch-p1 tip jail1
Configuring digi/ports
/etc/remote
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.
We have 2 digi's at castle we connect to:
#3-7 10.1.4.10 virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D: virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
and
#3-6 10.1.4.11 jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf: jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
The only things you need to edit are the first part (i.e. jail4) and the speed (i.e. 9600). You can decipher which port on the digi each line corresponds to by the CP001s or CO001s (port 1 on digi1 and digi2), CP002s or CO002s (port 2 on digi1 and digi2)
drpadmin
The tool you use to configure a device to a digi box is drpadmin:
bash-2.03$ su Password: # drpadmin Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s 0 10.1.4.10 32 CO 771 never 1027 1 10.1.4.11 32 CP 771 never 1027 2 65.116.11.2 8 el 771 never 1027 Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :
Use those commands above to modify the devices available.
Switching IP/hostname
Edit:
/etc/defaultrouter /etc/hosts /etc/hostname.hme0 /etc/nodename Maybe needed to run: # ifconfig hme0 10.1.4.4 up
devweb
We do web development on devweb.johncompanies.com
Currently this is a jail running on jail17 / 69.55.230.8
If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart
All website development work should be done here first. It works exactly like and is setup like our main site.