Latest revision as of 15:05, 3 October 2023

jails[edit]

jail1[edit]

Location: castle, SHUTDOWN
OS: FreeBSD 6.2 i386
Networking: Priv IP: 10.1.4.101 (PCI nic), Pub IP: 69.55.230.107 (onboard)
Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
Remote management: none
Disk accounting: gvinum

jail2[edit]

Location: castle, cab 6-16
OS: FreeBSD 7.2 amd64
Networking: Priv IP: 10.1.4.102, Pub IP: 69.55.228.53 (2 onboard nics)
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 146 GB (2 x 146GB) RAID1 array, two 300 GB (4 x 300GB) RAID1 arrays running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
Remote management: DRAC @ 10.1.4.232
Disk accounting: md

jail3[edit]

Location: I2b SHUTDOWN
OS: FreeBSD 8.3 amd64
Networking: Priv IP: 10.1.2.103, Pub IP: 69.55.229.7 (2 onboard nics)
Hardware: Supermicro (custom build). 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 300 GB (2 x 300GB) RAID1 array running on a 3ware 8006-2LP RAID card.
Remote management: none
Disk accounting: md

Notes[edit]

We should not add users to this server since it is at I2B
must be ssh'd to from nat2
is a super jail for customer col01737

jail4[edit]

Location: castle, cab 6-17
OS: FreeBSD 9.1 x86_64
Networking: Priv IP: 10.1.4.104, Pub IP: 69.55.228.104 (2 onboard nics)
Hardware: Dell 2850. 6 x 300GB SCSI drives (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
CPU: 2 x Intel(R) Xeon(TM) CPU 2.80GHz (8 virtual CPUs)
RAM: 16 GB ( 4 x 4GB Reg ECC )
Drives: one 1.4 TB RAID 5 array (6 x 300GB SCSI) Dell-branded (PERC 4e)LSI megarc RAID card.
Remote management: None
Disk accounting: md

Notes[edit]

Only FreeBSD 9.1 jail Not upgraded to FBSD 9.2 or 9.3 because too many libraries modified (would require customers to rebuild apps).

jail5[edit]

Location: castle, cab 3-6
OS: FreeBSD 10.1 x86_64
Networking: Priv IP: 10.1.4.105, Pub IP: 69.55.230.105 (2 onboard nics)
Hardware: Supermicro JC-14004 - Intel S1200BTL motherboard - 6 SATA/SAS drive bays (2 colums of 3), Dual power supply.
CPU: 1 x Intel(R) Xeon(TM) E3-1230 V2 CPU 3.30 GHz (8 virtual CPUs)
RAM: 32 GB ( 4 x 8GB ECC )
Drives: 1x80 GB SATA SSD on motherboard + one 2.6 TB RAID 5 array 4x1 TB + 3ware 9650 RAID card.
Remote management: Intel RMM 4 - 10.1.4.235
Disk accounting: md

Notes[edit]

Only FreeBSD 10.1 jail used for bhyve virtuals.

Use ~+Ctrl-D to disconnect from console (vm attach colXXXXX).

jail6[edit]

Location: castle, cab 6-16
OS: FreeBSD 10.3 x86_64
Networking: Priv IP: 10.1.4.106, Pub IP: 69.55.230.106 (2 onboard nics)
Hardware: Supermicro JC-14004 - Intel S1200BTL motherboard - 6 SATA/SAS drive bays (2 colums of 3), Dual power supply.
CPU: 1 x Intel(R) Xeon(TM) E3-1230 V2 CPU 3.30 GHz (8 virtual CPUs)
RAM: 32 GB ( 4 x 8GB ECC )
Drives: one 2.7 TB ZFS RAID 6 array 5x1 TB
Remote management: Intel RMM 4 - 10.1.4.236
Disk accounting: zfs

jail7[edit]

Location: castle, cab 3-5 SCHEUDLED SHUTDOWN 9/30/19
OS: FreeBSD 6.3 i386
Networking: Priv IP: 10.1.4.107, Pub IP: 69.55.230.108 (2 onboard nics)
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 146 GB (4 x 146GB) RAID1 arrays, one 74 GB (2 x 74GB) RAID1 array running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
Remote management: DRAC @ 10.1.4.237
Disk accounting: gvinum

Notes[edit]

Do not run a verify while OS/jails running, will crash.

jail8[edit]

Location: castle, cab 3-6
OS: FreeBSD 8.0 amd64
Networking: Priv IP: 10.1.4.108, Pub IP: 69.55.234.2 (2 onboard nics)
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
Remote management: DRAC @ 10.1.4.238
Disk accounting: md

jail9[edit]

Location: castle, cab 3-6
OS: FreeBSD 8.2 amd64
Networking: Priv IP: 10.1.4.109, Pub IP: 69.55.232.36 (2 onboard nics)
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 300GB) RAID1 array running on an LSI-based, Dell-branded (PERC 5/i) RAID card.
Remote management: DRAC @ 10.1.4.239
Disk accounting: md

jail11[edit]

Location: castle, cab 3-7
OS: FreeBSD 4.7 i386
Networking: Priv IP: 10.1.4.111 (PCI nic), Pub IP: 69.55.236.92 (onboard)
Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
Remote management: none
Disk accounting: vinum

mx1[edit]

Location: castle, SHUTDOWN AND SCRAPPED
OS: FreeBSD 4.11 i386
Networking: Priv IP: 10.1.4.201 (PCI nic), Pub IP: 69.55.237.3 (onboard)
Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array running on an Adaptec-based, Dell-branded (perc) RAID card.
Remote management: none
Disk accounting: vinum

Notes[edit]

is our (old) backup mail/dns vps service host

mx2[edit]

Location: castle, SHUTDOWN AND SCRAPPED
OS: FreeBSD 7.1 i386
Networking: Priv IP: 10.1.4.202 (PCI nic), Pub IP: 69.55.237.90 (onboard)
Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
Disk accounting: gvinum

Notes[edit]

is our latest backup mail/dns vps service host

jail17[edit]

Location: castle, cab 3-7
OS: FreeBSD 4.10 i386
Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
Remote management: none
Disk accounting: vinum
Host of devweb.johncompanies.com and www.utopian.com/mail.utopian.com

jail18[edit]

Location: castle, cab 3-5 SCHEDULED SHUTDOWN 9/30/19
OS: FreeBSD 4.10 i386
Networking: Priv IP: 10.1.4.118 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
Remote management: none
Disk accounting: vinum
Host of ns2c.johncompanies.com (now on ns2c.johncompanies.com on ganeti)

jail19[edit]

Location: castle, cab 3-5 SCHEDULED SHUTDOWN 9/30/19
OS: FreeBSD 6.1 i386
Networking: Priv IP: 10.1.4.119 (PCI nic), Pub IP: 69.55.228.200 (onboard nics)
Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S RAID card.
Remote management: none
Disk accounting: gvinum

virts[edit]

quar1[edit]

Location: castle, SHUTDOWN AND SCRAPPED
OS: RedHat 7.3 x86
Networking: Priv IP: 10.1.4.151 (PCI nic), Pub IP: 69.55.227.2 (onboard nic)
Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array, running on an Adaptec-based, Dell-branded (perc) RAID card.
Remote management: none
Virtuozzo version: 2.6.1
VZ license: hwid=23C0.C0E1.6FDD.08BA.8971.8E1C.EBD5.1EDC serial=0DE6.903E.E239.E23F.470C.4369.4104.A5A4

Notes[edit]

used to be the home of customers who's VE's would just run out of control/badly
has a max of 10 VE's allowed to run

virt9[edit]

Location: castle, cab 3-7
OS: RedHat 7.3 x86
Networking: Priv IP: 10.1.4.59 (PCI nic), Pub IP: 69.55.226.161 (onboard nic)
Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 74 GB (2 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
Remote management: none
Virtuozzo version: 2.6.1
VZ license: hwid=BC15.B4D6.0D25.A5FE.F3BA.D518.E351.AE3F serial=F6AD.B6B4.5650.8869.C97C.73EE.AF65.FA8B

virt11[edit]

Location: castle, cab 3-6
OS: CentOS 5.4 x86
Networking: Priv IP: 10.1.4.61, Pub IP: 69.55.238.3, 2 onboard nics
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
Remote management: DRAC @ 10.1.4.211
Virtuozzo version: 4.0.0
VZ license: hwid="029D.A187.78E1.480F.49E3.E20A.7389.7F79" serial="163C.F3E2.195F.96B5.2D38.8937.9600.4A05" key_number="VZ.00172378.0006"

 vzlicload -p A40R00-D8CS00-5D8817-A2RB23-8Y9J78

virt12[edit]

Location: castle, cab 3-7
OS: CentOS 5.2 x86
Networking: Priv IP: 10.1.4.62, Pub IP: 69.55.227.70, 2 onboard nics
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 300 GB (2 x 300GB) RAID1 arrays one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
Remote management: DRAC @ 10.1.4.212
Virtuozzo version: 4.0.0
VZ license: hwid="0C53.A413.E095.B4F4.51BC.D740.6919.A77B" serial="84E5.9498.3759.E683.E24B.2514.CA72.DC31"

virt13[edit]

Location: castle, cab 6-17
Switch port: P13-
OS: CentOS 6.2 x86_64
Networking: Priv IP: 10.1.4.63, Pub IP: 69.55.226.2, 2 onboard nics
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
CPU: 2 x Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (8 virtual cores)
RAM: 32 GB (8 x 4GB DDR2 FB-DIMM ECC 667MHz)
Drives: one 146 GB (2 x 146GB) RAID1 array, one 600 GB (2 x 600GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
Remote management: DRAC @ 10.1.4.213
Virtuozzo version: 4.7.0
VZ license: hwid="7D07.93BE.0B1F.7D2B.B039.4B5B.48B6.453B" serial="60A4.A94C.44BB.DCD6.8D03.1778.605B.10FE"

Notes[edit]

home to our latest/current signups
currently the only 64bit vz host

virt14[edit]

Location: castle, cab 6-16
Switch Port: p13-
OS: CentOS 6.4 x86_64
Networking: Priv IP: 10.1.4.64 Pub IP: 69.55.225.14 2 onboard nics
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
CPU: 2 x Xeon 5140 Dual Core @ 2.33GHz (4 virtual CPUs)
RAM: 32 GB (8 x 4GB Reg ECC)
Drives: one 146 GB (2 x 146 GB SAS) RAID1 array, and one 1TB RAID1 array (2 x 1 TB SATA), running on an LSI-based, Dell-branded (perc 5/i) RAID card.
Remote management: DRAC @ 10.1.4.214
Virtuozzo version: 4.7.0
VZ license: hwid="EA32.2CA0.2368.F5FC.DFBE.6724.5AC0.8ED0" serial="DA0D.F464.0BCE.35B8.C0C0.28B6.D921.F3FD" key_number="VZ.02634184.0070"
Activation Key="A00E00-A0WC02-W1A863-482N41-BQAY84"

Notes[edit]

our latest virt
Temp server to offload Virt13 till we can get a Cloud going.
virt 13 and 14 currently the only 64bit vz hosts

virt15[edit]

Location: SHUTDOWN
OS: RedHat 9 x86
Networking: Priv IP: 10.1.4.65, Pub IP: 69.55.232.160 (2 onboard nics)
Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 74 GB (2 x 74GB) RAID1 array, two 146 GB (2 x 146GB) RAID1 arrays, running on an LSI MegaRAID SCSI 320-1 RAID card.
Remote management: none
Virtuozzo version: 2.6.2
VZ license: hwid=A90F.6F48.E723.D8BA.3025.184A.5B73.D11E serial=E94B.5164.C1E6.A67F.67D1.7D96.0B6C.5524

virt16[edit]

Location: castle, cab 3-7
OS: Fedora Core 4 x86
Networking: Priv IP: 10.1.4.66, Pub IP: 69.55.232.2 (2 onboard nics)
Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI MegaRAID SCSI 320-1 RAID card.
Remote management: none
Virtuozzo version: 3.0.0
VZ license: hwid=DEFA.A325.7230.BBC8.9715.8B52.3FD7.27BE serial=66C0.41EA.3FBB.11D3.9CC6.55C7.09AE.14AB

virt17[edit]

Location: castle, cab 3-6
OS: CentOS 4.4 x86
Networking: Priv IP: 10.1.4.67, Pub IP: 69.55.232.162, 2 onboard nics
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 146 GB (2 x 146GB) RAID1 arrays running on an LSI-based, Dell-branded (perc 5/i) RAID card.
Remote management: DRAC @ 10.1.4.217
Virtuozzo version: 3.0.0
VZ license: hwid=2E14.AED9.70B8.C26E.D99F.B0D3.BCD2.229C serial=2A11.DAD0.61DB.E889.8DF4.9AF7.CF82.3C37

virt19[edit]

Location: castle, cab 3-6
OS: CentOS 5.2 x86
Networking: Priv IP: 10.1.4.69, Pub IP: 69.55.236.2, 2 onboard nics
Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, running on an LSI-based, Dell-branded (perc 5/i) RAID card.
Remote management: DRAC @ 10.1.4.219
Virtuozzo version: 3.0.0
VZ license: hwid=3968.13F7.B2AC.8952.8E19.13A9.6EF5.5822 serial=061D.84CD.CCE5.B213.15B5.C061.D6A7.B034

mail[edit]

Summary[edit]

This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

Location: castle, cab 3-7
OS: FreeBSD 4.10 x86
Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

Services Provided[edit]

mail
web
mysql
bigbrother server/pager
snmp
named in a jail (ns1c)

email[edit]

This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:

dave
linux
support
payments
sales
tech1
info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:

# more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
  {
   :0h
   FROMANDSUBJECT=|formail -XFrom: -XSubject:

   :0fwh
   | /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

   :0
  ! -t
  }

control: cd /etc/mail; make stop (stop), cd /etc/mail; make start (start)

The following aliases are also in place:

debian:         linux
jobs:   info
careers:        info
#reboot:         6128102202@txt.att.net
#reboot:         8582298897@vtext.com
reboot:         pager
#pager: 8582298897@vtext.com
pager:  4158718324@txt.att.net
tech1on:  "| /usr/local/sbin/tech1on.sh"
tech1off:  "| /usr/local/sbin/tech1off.sh"

To change them, edit /etc/aliases and then run newaliases

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

~support/tech1on.sh

To disable

~support/tech1off.sh

Or via email:

tech1on@johncompanies.com tech1off@johncompanies.com

IP Blocking[edit]

01000 deny ip from 188.92.72.5 to any
01003 deny ip from any to 122.49.31.50
01004 deny ip from 122.49.31.50 to any
01014 deny ip from 74.208.225.225 to any
01015 deny ip from any to 216.243.118.35
01016 deny ip from 216.243.118.35 to any
01017 deny ip from any to 216.243.118.36
01018 deny ip from 216.243.118.36 to any
01020 deny ip from 112.215.0.0/18 to any   2014-08-13 Blocked PT Excelcomindo Pratama (Indonesia) for fradulent credit card attempts 
01020 deny ip from 112.215.64.0/20 to any  2014-08-13 Blocked PT Excelcomindo Pratama (Indonesia) for fradulent credit card attempts 
01022 deny ip from 120.168.0.0/24 to any   2014-08-13 Blocked Indosat 3G Broadband (Indonesia) for fradulent credit card attempts
01022 deny ip from 120.175.213.0/24 to any 2014-08-13 Blocked Indosat 3G Broadband (Indonesia) for fradulent credit card attempts

web[edit]

See Management System / Public Website / Signup

mysql[edit]

mysql 4.1.22 is running on port 3306

datadir: /mnt/data1/db/mysql/
config: /etc/my.cnf
database: jc
control: /usr/local/etc/rc.d/mysql-server.sh stop (stop), /usr/local/etc/rc.d/mysql-server.sh start (start)

bigbrother[edit]

There is a client running on mail (which monitors the services running on mail and mail itself), installed under /usr/home/bb/bbc1.9e-btf
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under /usr/home/bb/bbsrc/bb1.9i-btf

Both are running under the user bb

Refer to BigBrother for more about use.

DNS (ns1c.johncompanies.com)[edit]

ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from /mnt/data1/ns1c-dir

See DNS for more details

Usage and Notes[edit]

always mounted to backup1 and backup2 via nfs:

backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)

Cronjobs[edit]

* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1

Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/

Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg

Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done

Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack

Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup

Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update

Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*

Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl

Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl

Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl

Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl

Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl

Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl

Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done

Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl

Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl

Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl

Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl

Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl

Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl

Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null

Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys

Backup data on isys

Regular maintenance[edit]

Check RAID array

Building a new Mail Server[edit]

Installations[edit]

I used FreeBSD 11.2

The order is important especially for the Web Server.

Web Server[edit]

I used FreeBSD 11.2

 perl 5.26 
 OpenSSL 1.0.2o-freebsd
 pcre
 apache22 
 mod_perl2
 PayflowPro
 mariadb 55 server and client

Installation order is important

install perl 5.26.2 from ports

<ore> cd /usr/ports/lang/perl5.26/ make [X] PERL_64BITINT Use 64 bit integers (on i386) [X] USE_PERL Rewrite links in /usr/bin (the rest unchecked make install

install OpenSSL 1.0.2o-freebsd

cd /usr/ports/
make install

install pcre

cd /usr/ports/
make install

install Apache22

cd /usr/ports/distfiles
fetch http://archive.apache.org/dist/httpd/httpd-2.2.32.tar.gz


cd /usr/ports/www/apache22/tmp
fetch --no-verify-peer http://mirror.nexcess.net/apache//httpd/httpd-2.2.34.tar.gz
tar xvzf httpd-2.2.34.tar.gz
./configure --prefix=/usr/local/apache --with-ssl=/usr/local/openssl/ --enable-ssl --enable-so --with-mpm=prefork --enable-threads --enable-mods-shared='mime alias setenvif dir' --enable-modules='mime alias setenvif dir' --with-pcre=/usr/local
make install
apachectl restart

cd /usr/ports/www/apache22
echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf
make DISABLE_VULNERABILITIES=yes
make install

install mod_perl2

cd /usr/ports/www/mod_perl2
echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf
make DISABLE_VULNERABILITIES=yes
make install

install mariadb

cd /usr/ports/databases/mariadb-103-server
echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf
make DISABLE_VULNERABILITIES=yes
make install

Mail Server[edit]

I used Postfix for email

DNS Server (ns1c.johncompanies.com)[edit]

ns2c[edit]

Summary[edit]

Location: castle, on lamphost ganeti cloud
OS: FreeBSD 11.2 x86_64
Networking: Pub IP: 69.55.230.3 Private access: gnt-instance console ns2c.johncompanies.com
Hardware: on ganeti cloud gn6.jcihosting.net secondary gn1.jcihosting.net
CPU: 1
RAM: 1 GB
Drives: 10 GB
Remote management: gnt-instance console ns2c.johncompanies.com

nat[edit]

Summary[edit]

This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

Location: castle, cab 3-7
OS: FreeBSD 9.1 i386
Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
Hardware: Custom 1U. single power supply.
Drives: one 8 GB IDE drive

Services Provided[edit]

nat

nat control[edit]

All rules are contained in and look like:

cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32

A simple entry looks like:

bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32

Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:

ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

nat2[edit]

Summary[edit]

This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

Location: i2b, cab 6
OS: FreeBSD 6.4 x86
Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

Services Provided[edit]

nat
bigbrother
ns3c (jail)
wiki (jail)
ntp

nat config[edit]

Here's what's currently nat'd on nat2:

cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

build[edit]

partition map:

/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~

edit /etc/make.conf

echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

turn off all ttyv's except 0 and 1 in /etc/ttys

also turn on ttyd0, change type to vt100:

vi /etc/ttys

ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0   "/usr/libexec/getty std.9600"   vt100   on  secure

kill -1 1

on console server:

vi /etc/remote

(rename port to jail8 depending on where and which digi plugged into) test serial console

populate hosts

echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

put key in authorized_keys on backup3

cd
ssh-keygen -t dsa -b 1024

(default location, leave password blank)

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

edit root's path and login script:

vi /root/.cshrc

Change alias entries (add G):

alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:

set prompt = "`/bin/hostname -s` %/# "

install cvsup

cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

get latest sources for this release:

cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

configure new kernel.

cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld 
mergemaster -i

populate /etc/rc.conf with IPs and NFS settings

vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"

reboot. Confirm new kernel is loaded

uname -a

update ports:

cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

Install raid mgmt tool

cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli

Test:

./tw_cli info c0

install rsync from ports

cd /usr/ports/net/rsync
make install clean

choose default options

install perl from ports

cd /usr/ports/lang/perl5.8
make install clean

install screen from ports

cd /usr/ports/sysutils/screen
make install clean

install bb client

adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="nat2,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
        $1 $TOPARGS > $BBTMP/TOP.$$
#        /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

configure bb on mail:

vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

configure ntp

echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf

/usr/sbin/ntpd -A -p /var/run/ntpd.pid 
sleep 2; ntpq -p

(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

fwd and reverse lookups on ns1c

vr johncompanies.com
(edit the PTR too)

setup backups, nfs mount

mkdir /backup3
echo 'backup3:/data           /backup3        nfs     rw,bg           0       0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3: setup backup dirs:

ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to

vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup 
vi /usr/local/etc/rsync.backup 
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

edit sshd_config for security

vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
    if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

netflow stuff

add crontab entries

crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; 
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

add nat rules

vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules

shell for user

cp /root/.cshrc ~user/
vi ~user/

change # to $

mrtg

cd /usr/ports/net-mgmt/mrtg
make install clean

(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null

snmp firewall block

cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh

bwdb[edit]

Summary[edit]

This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

Location: castle, cab 3-7
OS: FreeBSD 4.10 x86
Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
Hardware: Custom 1U. Single power supply.
Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

Services Provided[edit]

netflow
mysql
bigbrother
snmp

netflow[edit]

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under /usr/home/flows/ (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)

1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue: /usr/home/working

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:

2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

mysql[edit]

The database storing all the traffic data is named traffic Tables:

mysql> show tables;
+---------------------------+
| Tables_in_traffic         |
+---------------------------+
| dailyIpTotals_69_55_224   |
| dailyIpTotals_69_55_225   |
| dailyIpTotals_69_55_226   |
| dailyIpTotals_69_55_227   |
| dailyIpTotals_69_55_228   |
| dailyIpTotals_69_55_229   |
| dailyIpTotals_69_55_230   |
| dailyIpTotals_69_55_231   |
| dailyIpTotals_69_55_232   |
| dailyIpTotals_69_55_233   |
| dailyIpTotals_69_55_234   |
| dailyIpTotals_69_55_235   |
| dailyIpTotals_69_55_236   |
| dailyIpTotals_69_55_237   |
| dailyIpTotals_69_55_238   |
| dailyIpTotals_69_55_239   |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224        |
| ipTotals_69_55_225        |
| ipTotals_69_55_226        |
| ipTotals_69_55_227        |
| ipTotals_69_55_228        |
| ipTotals_69_55_229        |
| ipTotals_69_55_230        |
| ipTotals_69_55_231        |
| ipTotals_69_55_232        |
| ipTotals_69_55_233        |
| ipTotals_69_55_234        |
| ipTotals_69_55_235        |
| ipTotals_69_55_236        |
| ipTotals_69_55_237        |
| ipTotals_69_55_238        |
| ipTotals_69_55_239        |
| portTotals_69_55_224      |
| portTotals_69_55_225      |
| portTotals_69_55_226      |
| portTotals_69_55_227      |
| portTotals_69_55_228      |
| portTotals_69_55_229      |
| portTotals_69_55_230      |
| portTotals_69_55_231      |
| portTotals_69_55_232      |
| portTotals_69_55_233      |
| portTotals_69_55_234      |
| portTotals_69_55_235      |
| portTotals_69_55_236      |
| portTotals_69_55_237      |
| portTotals_69_55_238      |
| portTotals_69_55_239      |
+---------------------------+

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field     | Type        | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id        | varchar(23) |      | PRI |         |       |
| date      | date        | YES  |     | NULL    |       |
| ip        | varchar(15) | YES  | MUL | NULL    |       |
| direction | tinyint(1)  | YES  |     | NULL    |       |
| octets    | bigint(12)  | YES  |     | NULL    |       |
| packets   | int(11)     | YES  |     | NULL    |       |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
       id: 6955224194-20100917-1
     date: 2010-09-17
       ip: 69.55.224.194
direction: 1
   octets: 8821
  packets: 91

The id is a unique identifier (key), direction indicates incoming or outgoing traffic (outbound = 2, inbound = 1), octets are the amount of traffic in kilobytes, and packets is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field     | Type       | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date      | datetime   | YES  |     | NULL    |       |
| ip        | char(15)   | YES  | MUL | NULL    |       |
| direction | tinyint(1) | YES  |     | NULL    |       |
| octets    | bigint(20) | YES  |     | NULL    |       |
| packets   | int(11)    | YES  |     | NULL    |       |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
     date: 2010-01-11 19:30:00
       ip: 69.55.224.13
direction: 1
   octets: 288
  packets: 6
*************************** 2. row ***************************
     date: 2010-01-11 19:30:00
       ip: 69.55.224.12
direction: 1
   octets: 216
  packets: 4

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field     | Type        | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id        | varchar(28) |      | PRI |         |       |
| date      | date        | YES  |     | NULL    |       |
| ip        | varchar(15) | YES  | MUL | NULL    |       |
| direction | tinyint(1)  | YES  |     | NULL    |       |
| protocol  | smallint(3) | YES  |     | NULL    |       |
| port      | int(11)     | YES  |     | NULL    |       |
| octets    | bigint(11)  | YES  |     | NULL    |       |
| packets   | int(11)     | YES  |     | NULL    |       |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
       id: 695522496-20091218-1-6-23
     date: 2009-12-18
       ip: 69.55.224.96
direction: 1
 protocol: 6
     port: 23
   octets: 1796
  packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
     date: 2010-09-07 18:45:00
       ip: 69.55.224.254
direction: 1
 protocol: 6
     port: 99999
   octets: 144
  packets: 3

This is largely the same with 2 more additions: protocol (1=ICMP, 6=TCP, 17=UDP), and port which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

Regular maintenance[edit]

Check RAID array
archive data from database

archive_daily.pl 2012 09

This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09

This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

if space becomes tight, move flow files and exported data to a backup server, both located in /usr/home/flowbin/archive and /usr/home/exported, respectively

Slaving[edit]

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:

bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3

on slave: in my.cnf:

server-id       = 2
master-host     =   10.1.4.203
master-user     =   repl
master-password =   qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%

on master:

touch /usr/home/working/.lock

(make sure processflows not running)

FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File            | Position  | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 |              |                  |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/

on slave:

mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;

cd /usr/home/database/traffic
scp *

optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;

Build[edit]

BIOS Config[edit]

disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

Install OS[edit]

Install FreeBSD 8.3 amd64

partition map:

/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~

edit /etc/make.conf

Castle:

echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

i2b:

echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

turn off all ttyv's except 0 and 1 in /etc/ttys

also turn on ttyu0, change type to vt100:

vi /etc/ttys

ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0   "/usr/libexec/getty std.9600"   vt100   on secure

kill -1 1

on console server:

vi /etc/remote

(rename port to jail8 depending on where and which digi plugged into) test serial console

populate hosts

i2b:

echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

castle:

echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts

put key in authorized_keys on backup1 and backup2

cd
ssh-keygen -t dsa -b 1024

(default location, leave password blank)

castle:

cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:

cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

edit root's path and login script:

vi /root/.cshrc

Change alias entries (add G):

alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG

and alter the prompt, set the following:

set prompt = "`/bin/hostname -s` %/# "

install cvsup

cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

get latest sources for this release:

cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null

configure new kernel

cd /usr/src/sys/amd64/conf 
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:

vi bwdb
ident  bwdb

build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld 
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

populate /etc/rc.conf with IPs and NFS settings

castle:

vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"

i2b:

vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"

reboot. Confirm new kernel is loaded

uname -a

update ports:

cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null

Install raid mgmt tool

cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli

Test:

./tw_cli info c0

Grab raid check script:

scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:

crontab -e
*/5 * * * * /usr/local/etc/3wraidchk

install rsync from ports

cd /usr/ports/net/rsync
make install clean

choose default options

install perl from ports

cd /usr/ports/lang/perl5.8
make install clean

choose default options

install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:

rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:

echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 

MACHINE="bwdb,johncompanies,com"      # HAS TO BE IN A,B,C FORM

Edit for machine name.

Have bb watch for flow-capture, mysql

cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT 
(look for errors)
exit

Put in script to start bb @ boot:

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

configure bb on mail

vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

configure ntp server

Castle:

echo "server 10.1.4.1" > /etc/ntp.conf

I2b:

echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p

(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

fwd and reverse lookups on ns1c

vr johncompanies.com

(edit the PTR too)

setup backups

echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config

Castle: setup backup dirs:

ssh backup1 mkdir -p /data/bwdb/current

on backup1, add the system to

vi /usr/local/sbin/snapshot_rotate

I2b: setup backup dirs:

ssh backup3 mkdir -p /data/bwdb/current

on backup3, add the system to

vi /usr/local/sbin/snapshot_archive

Copy over the backup script:

scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change config var to point to correct config file location: /usr/local/etc/backup.config

crontab -e
5 0 * * * /usr/local/etc/rsync.backup

make /root/logs

mkdir /root/logs

edit sshd_config for security

vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`

Edit for private IP.

snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)

cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity  jcread 10.1.4.5
rocommunity  jcread 10.1.4.202

netflow[edit]

Install flow tools:

cd /usr/ports/net-mgmt/flow-tools
make install clean

Defaults.

mkdir /usr/home/flows

Flow start script:

echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:

cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh

Edit for private IP.

Confirm netflow is running after running scripts:

newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
  config     get or set configuration of node at <path>
  connect    Connects hook <peerhook> of the node at <relpath> to <hook>
  debug      Get/set debugging verbosity level
  dot        Produce a GraphViz (.dot) of the entire netgraph.
  help       Show command summary or get more help on a specific command
  list       Show information about all nodes
  mkpeer     Create and connect a new node to the node at "path"
  msg        Send a netgraph control message to the node at "path"
  name       Assign name <name> to the node at <path>
  read       Read and execute commands from a file
  rmhook     Disconnect hook "hook" of the node at "path"
  show       Show information about the node at <path>
  shutdown   Shutdown the node at <path>
  status     Get human readable status information from the node at <path>
  types      Show information about all installed node types
  write      Send a data packet down the hook named by "hook".
  quit       Exit program
+ show netflow:
  Name: netflow         Type: netflow         ID: 00000004   Num hooks: 3
  Local hook      Peer name       Peer type    Peer ID         Peer hook
  ----------      ---------       ---------    -------         ---------
  export          <unnamed>       ksocket      00000005        inet/dgram/udp
  out0            em0             ether        00000001        upper
  iface0          em0             ether        00000001        lower
+

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh

process flow tools[edit]

mkdir /usr/home/flowbin
mkdir /usr/home/working

Install modules:

cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean

Queue script:

cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
    push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
    my $newdate = date::AddDays($date, -1);
    my $flowdir = mkFlowDir($newdate);
    `mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
    my $date = shift;
    $date =~ /([0-9]{4}-[0-9]{2})/;
    my $yearmonth = $1;
    return "$flowbase/$yearmonth/$date";
}

Date.pm module:

cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
#      e-Monitoring Networks, Inc.  All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
    my $date1 = shift;
    my $date2 = shift;
    my ($year1, $month1, $day1) = &DateToymd($date1);
    my ($year2, $month2, $day2) = &DateToymd($date2);
    my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
    return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
    my $date = shift;
    my $days = shift;
    my ($year, $month, $day) = &DateToymd($date);
    my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
    my $ndate = &ymdToDate($nyear, $nmonth, $nday);
    return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
    my $date = shift;
    my $time = shift;
    my $addhours = shift;
    my $adddays = 0;
    if (abs($addhours / 24) >= 1) {
        $adddays = int($addhours / 24);
        $addhours -= $adddays * 24;
    }
    my ($year, $month, $day) = &DateToymd($date);
    my ($hour, $minute, $second) = &TimeTohms($time);
    my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
                                                          $hour, $minute, $second,
                                                          $adddays, $addhours, 0, 0);
    my $ndate = &ymdToDate($ny, $nm, $nd);
    my $ntime = &hmsToTime($nh, $nmin, $ns);
    return $ndate, $ntime;
}

sub AddMinutes {
    my $date = shift;
    my $time = shift;
    my $minutes = shift;
    my ($year, $month, $day) = &DateToymd($date);
    my ($hour, $minute, $second) = &TimeTohms($time);
    my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
                                                          $hour, $minute, $second,
                                                          0, 0, $minutes, 0);
    my $ndate = &ymdToDate($ny, $nm, $nd);
    my $ntime = &hmsToTime($nh, $nmin, $ns);
    return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
    my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
    my $date = &ymdToDate($y, $m, $d);
    my $time = &hmsToTime($h, $min, $s);
    return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
    my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
    return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
    my $date = shift;
    if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
        my $day = $3;
        my $month = $2;
        my $year = $1;
        return $year, $month, $day;
    }
    return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
    my $time = shift;
    if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
        my $hour = $1;
        my $minute = $2;
        my $second = $3;
        if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
        if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
        if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
        return $hour, $minute, $second;
    }
    return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
    my $year = shift;
    my $month = shift;
    my $day = shift;
    if (defined($year) && defined($month) && defined ($day)) {
        $month = sprintf("%02d", $month);
        $day = sprintf("%02d", $day);
        return "$year-$month-$day";
    }
    return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
    my $hour = shift;
    my $minute = shift;
    my $second = shift;
    if (defined($hour) && defined($minute) && defined ($second)) {
        if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
        if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
        if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
        return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
    }
    return undef;
}

sub CompareDates { #compares two date and time pairs
    my $date1 = shift;
    my $time1 = shift;
    my $date2 = shift;
    my $time2 = shift;

    my ($year1, $month1, $day1) = &DateToymd($date1);
    my ($hour1, $minute1, $second1) = &TimeTohms($time1);
    my ($year2, $month2, $day2) = &DateToymd($date2);
    my ($hour2, $minute2, $second2) = &TimeTohms($time2);

#    &debug("$year1, $month1, $day1, $year2, $month2, $day2");
    my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
    if ($days > 0) { return 1;}
    if ($days < 0) { return -1;}
    if ($days == 0) { #same day, compare times
        my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
        my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
        if ($seconds1 < $seconds2) { return 1;}
        if ($seconds1 > $seconds2) { return -1;}
        if ($seconds1 == $seconds2) { return 0;}
    }
    return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
    my $date1 = shift;
    my $time1 = shift;
    my $date2 = shift;
    my $time2 = shift;
    my $window = shift;

    my ($year1, $month1, $day1) = &DateToymd($date1);
    my ($hour1, $minute1, $second1) = &TimeTohms($time1);
    my ($year2, $month2, $day2) = &DateToymd($date2);
    my ($hour2, $minute2, $second2) = &TimeTohms($time2);

    my ($day, $hour, $minute, $second) =
        &Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
                    $year2, $month2, $day2, $hour2, $minute2, $second2);
    $minute *= 60;
    $hour *= 3600;
    $day *= 86400;
    my $total = $second + $minute + $hour + $day;
    if (abs($total) < $window) {
        return 1;
    }
    return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
    my $date1 = shift;
    my $time1 = shift;
    my $date2 = shift;
    my $time2 = shift;
    my $date3 = shift;
    my $time3 = shift;
    if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
        return 0;
    }
    if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
        return 0;
    }
    return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
    my $date = shift;
    my $time = shift;
    my ($year, $month, $day) = &DateToymd($date);
    my ($hour, $minute, $second) = &TimeTohms($time);
    my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
                                      $year, $month, $day, $hour, $minute, $second);
    my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
    return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
    my $seconds = shift;
    my $days = int($seconds / 86400);
    $seconds -= $days * 86400;
    my $hours = int($seconds / 3600);
    $seconds -= $hours * 3600;
    my $minutes = int($seconds / 60);
    $seconds -= $minutes * 60;
    my ($year, $month, $day, $hour, $minute, $second) =
        &Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
    $month = sprintf("%02d", $month);
    $day = sprintf("%02d", $day);
    $hour = sprintf("%02d", $hour);
    $minute = sprintf("%02d", $minute);
    $second = sprintf("%02d", $second);
    return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
    my $date = shift;
    my ($year, $month, $day) = &DateToymd($date);
    my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
    $name =~ /^[A-Za-z]{3}/;
    $name = $&;
    return $name;
}

sub ValiDate {
    return @_;



}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
    my $date = shift;
    my ($year, $month, $day) = &DateToymd($date);
    if (Day_of_Week($year,$month,$day) < 6) { return 1; }
    else { return 0; }
}

1; #don't remove this line

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:

crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

flow processing: i2b[edit]

cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
    push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
    if (-e "$queuedir/.lock") {
        open(FILE, "$queuedir/.lock");
        my $pid = <FILE>;
        chomp($pid);
        close(FILE);
        if (kill(0, $pid)) {
            #another process is using the queue, bail out
            exit(0);
        }
        else {
            #dead lock file, remove it
            `rm $queuedir/.lock`;
        }
    }
    open(FILE, "> $queuedir/.lock");
    print FILE "$$\n";
    close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
    unless($file =~ /^\./) {
        $file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
        my $date = "$1 $2:$3:$4";
        my $outfile = "$1-$2:$3.sql";
        unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
        my $condensedDate = $1;
        $condensedDate =~ s/-//g;
        my $iptotal = {};
        my $protototal = {};
        my $porttotal = {};

        &debug("started file $file at ");
        &debug(`date`);
        &debug("getting raw flow data (flow-print)");
        `cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
        &debug("aggregating data at ");
        &debug(`date`);
        unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
        LOOP: while (my $line = readline DATA) {
            my @d = split /[\s]+/, $line;
            if ($d[0] ne '' && $d[0] ne 'Start') {
                my $addr = 0;
                my $port = 0;

                #Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
                #0     1   2   3            4    5   6            7    8 9  10   11
                          #|
                          # outbound = 2, inbound = 1

                my (@src_ip) = split '\.', $d[3];
                my (@dst_ip) = split '\.', $d[6];

                if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
                   $d[2] = 2;
                   # hack for outbound bulk traffic counted 2x
                   #if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
                }
                # note- this is where we filter out IPs only found at i2b
                elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
                   $d[2] = 1;
                }
                else { next LOOP; }

                if ($d[2] == 2) {
                    $addr = $d[3];
                    # if the dst-port is low, store that
                    if ($d[7] <= 1024) { $port = $d[7]; }
                    # if the src-port is low, store that
                    elsif ($d[4] <= 1024) { $port = $d[4]; }
                    else { $port = 99999; }
                }
                elsif ($d[2] == 1) {
                    $addr = $d[6];
                    # if the dst-port is high, assume its return traffic, try to store src-port if low
                    if ($d[7] > 1024) {
                        if ($d[4] <= 1024) { $port = $d[4]; }
                        else { $port = 99999; }
                    } else {
                        $port = $d[7];
                    }
                } else {
                    next LOOP;
                }

                my (@ip) = split '\.', $addr;
                unless ($ip[0] == 69) { next LOOP; }
                unless ($ip[1] == 55) { next LOOP; }
                unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

                my $classC = "$ip[0]_$ip[1]_$ip[2]";

#                          IP        dir
#                if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
#                if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
#                if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
#                if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
#                if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
#                if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
                $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
                $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
            }
        }
        close(DATA);
        `rm /usr/home/working/tmp-$file`;
        &debug("processing ip totals at ");
        &debug(`date`);
        foreach my $classC (keys(%{$iptotal})) {
            my @values;
            foreach my $ip (keys(%{$iptotal->{$classC}})) {
                foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
                    my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
                    my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
#                    $packets = $packets > 2147483647 ? 0 : $packets;
                    if ($octets > 2147483647) {
                        my $ddir = $dir==1 ? 'in' : 'out';
                        #print SQL "$date $ip $ddir $octets\n";
#                        $octets = 0;
                    }
                    # dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
                    my $id = "$ip-$condensedDate-$dir";
                    $id =~ s/\.//g;
                    push @values, "('$date', '$ip', $dir, $octets, $packets)";
                    my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
                    print "$sql\n" if $dry;
                    print SQL "$sql;\n";
#                    $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
                }
            }

            # break inserts into 100 records at a time
            &debug("inserting $#values +1 values");
            while ($#values > 0) {
                my $sql = "insert into ipTotals_$classC values ";
                my $max_index = $#values > 100 ? 100 : $#values;
                for (my $i=0; $i<=$max_index; $i++) {
                    $sql .= shift @values;
                    $sql .= ',';
                }
                chop $sql;
                print "$sql\n" if $dry;
                print SQL "$sql;\n";
            }
        }

#        &debug("processing protocol totals at ");
#        &debug(`date`);
#        foreach my $classC (keys(%{$protototal})) {
#            $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
#            my @values;
#            foreach my $ip (keys(%{$protototal->{$classC}})) {
#                foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
#                    foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
#                        my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
#                        my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# #                        $octets = $octets > 2147483647 ? 0 : $octets;
# #                        $packets = $packets > 2147483647 ? 0 : $packets;
#                        # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
#                        my $id = "$ip-$condensedDate-$dir-$proto";
#                        $id =~ s/\.//g;
#                        push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
#                        my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
#                        print "$sql\n" if $dry;
#                        $db->query($sql) unless $dry;
# #                        $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
#                    }
#                }
#            }
#            $db->query("unlock tables") unless $dry;
#            my $sql = "insert into protoTotals_$classC values ";
#            $sql .= join ',', @values;
#            $db->query("lock tables protoTotals_$classC write") unless $dry;
#            print "$sql\n" if $dry;
#            $db->query($sql) unless $dry;
#            $db->query("unlock tables") unless $dry;
#        }

        &debug("processing port totals at ");
        &debug(`date`);
        foreach my $classC (keys(%{$porttotal})) {
            my @values;
            foreach my $ip (keys(%{$porttotal->{$classC}})) {
                foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
                    foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
                        foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
                            my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
                            my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
    #                        $octets = $octets > 2147483647 ? 0 : $octets;
    #                        $packets = $packets > 2147483647 ? 0 : $packets;

                            # dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
                            my $id = "$ip-$condensedDate-$dir-$proto-$port";
                            $id =~ s/\.//g;
                            push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
                            my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
                            print "$sql\n" if $dry;
                            print SQL "$sql;\n";

    #                        $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
                        }
                    }
                }
            }

            # break inserts into 100 records at a time
            &debug("inserting $#values +1 values");
            while ($#values > 0) {
                my $sql = "insert into portTotals_$classC values ";
                my $max_index = $#values > 100 ? 100 : $#values;
                for (my $i=0; $i<=$max_index; $i++) {
                    $sql .= shift @values;
                    $sql .= ',';
                }
                chop $sql;
                print "$sql\n" if $dry;
                print SQL "$sql;\n";
            }
        }

#                       12     1 8      1    1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
#                       12        1  8     1   1       3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
#                       12       1   8    1     1     5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
        #print "finished at ";
        #print `date`;
        `mv $queuedir/$file $archivedir` unless $dry;
        close(SQL);
        `bzip2 $sqldirworking/$outfile`;
        `mv $sqldirworking/$outfile.bz2 $sqldir/`;
    }
}
`rm $queuedir/.lock` unless $dry;

sub debug {
    my $message = shift;
    if ($debug) {
        print "$message\n";
    }
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07

This script sends the sql files to the traffic server for processing:

cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
    if (-e "$sqldir/.lock") {
        open(FILE, "$sqldir/.lock");
        my $pid = <FILE>;
        chomp($pid);
        close(FILE);
        if (kill(0, $pid)) {
            #another process is using the queue, bail out
            exit(0);
        }
        else {
            #dead lock file, remove it
            `rm $sqldir/.lock`;
        }
    }
    open(FILE, "> $sqldir/.lock");
    print FILE "$$\n";
    close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
   next unless $file =~ /bz2$/;

   my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
#   print "scp $sqldir/$file $remote:$sqldirremote";
   unless ($?==0) {
      push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
   }
   else {
      `mv $sqldir/$file $archive`;
      `ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
   }
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
   email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
    my $subj=shift;
    my $body=shift;
    use Mail::Sendmail;

    # prepare message
    my %mail = (
        To      => 'support@johncompanies.com,dave@johncompanies.com',
        From    => 'support@johncompanies.com',
        Subject => $subj,
        Message => $body,
        smtp    => 'mail.johncompanies.com',
    );
    sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
    my $message = shift;
    if ($debug) {
        print "$message\n";
    }
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07

crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

flow processing: castle[edit]

cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
    push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
    if (-e "$queuedir/.lock") {
        open(FILE, "$queuedir/.lock");
        my $pid = <FILE>;
        chomp($pid);
        close(FILE);
        if (kill(0, $pid)) {
            #another process is using the queue, bail out
            exit(0);
        }
        else {
            #dead lock file, remove it
            `rm $queuedir/.lock`;
        }
    }
    open(FILE, "> $queuedir/.lock");
    print FILE "$$\n";
    close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
    unless($file =~ /^\./) {
        $file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
        my $date = "$1 $2:$3:$4";
        my $condensedDate = $1;
        $condensedDate =~ s/-//g;
        my $iptotal = {};
        my $protototal = {};
        my $porttotal = {};

        &debug("started file $file at ");
        &debug(`date`);
        &debug("getting raw flow data (flow-print)");
        `cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
        &debug("aggregating data at ");
        &debug(`date`);
        unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
        LOOP: while (my $line = readline DATA) {
            my @d = split /[\s]+/, $line;
            if ($d[0] ne '' && $d[0] ne 'Start') {
                my $addr = 0;
                my $port = 0;

                #Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
                #0     1   2   3            4    5   6            7    8 9  10   11
                          #|
                          # outbound = 2, inbound = 1

                my (@src_ip) = split '\.', $d[3];
                my (@dst_ip) = split '\.', $d[6];

                if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
                    $src_ip[2] >= 224 && $src_ip[2] <= 239 &&
                    $src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
#                if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
                   $d[2] = 2;
                   # hack for outbound bulk traffic counted 2x
                   if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
                }
                elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
                       $dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
                       $dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
#                elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
                   $d[2] = 1;
                }
                else { next LOOP; }

                if ($d[2] == 2) {
                    $addr = $d[3];
                    # if the dst-port is low, store that
                    if ($d[7] <= 1024) { $port = $d[7]; }
                    # if the src-port is low, store that
                    elsif ($d[4] <= 1024) { $port = $d[4]; }
                    else { $port = 99999; }
                }
                elsif ($d[2] == 1) {
                    $addr = $d[6];
                    # if the dst-port is high, assume its return traffic, try to store src-port if low
                    if ($d[7] > 1024) {
                        if ($d[4] <= 1024) { $port = $d[4]; }
                        else { $port = 99999; }
                    } else {
                        $port = $d[7];
                    }
                } else {
                    next LOOP;
                }

                my (@ip) = split '\.', $addr;
                unless ($ip[0] == 69) { next LOOP; }
                unless ($ip[1] == 55) { next LOOP; }
                unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
#                unless ($ip[2] == 229) { next LOOP; }

                my $classC = "$ip[0]_$ip[1]_$ip[2]";

#                          IP        dir
#                if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
#                if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
#                if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
#                if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
#                if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
#                if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
                $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
                $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
            }
        }
        close(DATA);
        `rm /usr/home/working/tmp-$file`;
        &debug("processing ip totals at ");
        &debug(`date`);
        foreach my $classC (keys(%{$iptotal})) {
            $db->query("lock tables dailyIpTotals_$classC write") unless $dry;
            my @values;
            foreach my $ip (keys(%{$iptotal->{$classC}})) {
                foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
                    my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
                    my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
#                    $packets = $packets > 2147483647 ? 0 : $packets;
                    if ($octets > 2147483647) {
                        my $ddir = $dir==1 ? 'in' : 'out';
                        `echo "$date $ip $ddir $octets\n" >> $log`;
#                        $octets = 0;
                    }
                    # dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
                    my $id = "$ip-$condensedDate-$dir";
                    $id =~ s/\.//g;
                    push @values, "('$date', '$ip', $dir, $octets, $packets)";
                    my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
                    print "$sql\n" if $dry;
                    $db->query($sql) unless $dry;
#                    $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
                }
            }
            $db->query("unlock tables") unless $dry;

            $db->query("lock tables ipTotals_$classC write") unless $dry;
            # break inserts into 100 records at a time
            &debug("inserting $#values +1 values");
            while ($#values > 0) {
                my $sql = "insert into ipTotals_$classC values ";
                my $max_index = $#values > 100 ? 100 : $#values;
                for (my $i=0; $i<=$max_index; $i++) {
                    $sql .= shift @values;
                    $sql .= ',';
                }
                chop $sql;
                print "$sql\n" if $dry;
                $db->query($sql) unless $dry;
            }
            $db->query("unlock tables") unless $dry;
        }

        sleep 20;
#        &debug("processing protocol totals at ");
#        &debug(`date`);
#        foreach my $classC (keys(%{$protototal})) {
#            $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
#            my @values;
#            foreach my $ip (keys(%{$protototal->{$classC}})) {
#                foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
#                    foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
#                        my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
#                        my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# #                        $octets = $octets > 2147483647 ? 0 : $octets;
# #                        $packets = $packets > 2147483647 ? 0 : $packets;
#                        # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
#                        my $id = "$ip-$condensedDate-$dir-$proto";
#                        $id =~ s/\.//g;
#                        push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
#                        my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
#                        print "$sql\n" if $dry;
#                        $db->query($sql) unless $dry;
# #                        $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
#                    }
#                }
#            }
#            $db->query("unlock tables") unless $dry;
#            my $sql = "insert into protoTotals_$classC values ";
#            $sql .= join ',', @values;
#            $db->query("lock tables protoTotals_$classC write") unless $dry;
#            print "$sql\n" if $dry;
#            $db->query($sql) unless $dry;
#            $db->query("unlock tables") unless $dry;
#        }

        &debug("processing port totals at ");
        &debug(`date`);
        foreach my $classC (keys(%{$porttotal})) {
            $db->query("lock tables dailyPortTotals_$classC write") unless $dry;
            my @values;
            foreach my $ip (keys(%{$porttotal->{$classC}})) {
                foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
                    foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
                        foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
                            my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
                            my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
    #                        $octets = $octets > 2147483647 ? 0 : $octets;
    #                        $packets = $packets > 2147483647 ? 0 : $packets;

                            # dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
                            my $id = "$ip-$condensedDate-$dir-$proto-$port";
                            $id =~ s/\.//g;
                            push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
                            my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
                            print "$sql\n" if $dry;
                            $db->query($sql) unless $dry;
    #                        $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
                        }
                    }
                }
            }
            $db->query("unlock tables") unless $dry;

            $db->query("lock tables portTotals_$classC write") unless $dry;
            # break inserts into 100 records at a time
            &debug("inserting $#values +1 values");
            while ($#values > 0) {
                my $sql = "insert into portTotals_$classC values ";
                my $max_index = $#values > 100 ? 100 : $#values;
                for (my $i=0; $i<=$max_index; $i++) {
                    $sql .= shift @values;
                    $sql .= ',';
                }
                chop $sql;
                print "$sql\n" if $dry;
                $db->query($sql) unless $dry;
            }
            $db->query("unlock tables") unless $dry;
            sleep 10;
        }

#                       12     1 8      1    1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
#                       12        1  8     1   1       3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
#                       12       1   8    1     1     5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
        #print "finished at ";
        #print `date`;
        `mv $queuedir/$file $archivedir` unless $dry;
    }
}
`rm $queuedir/.lock` unless $dry;

sub debug {
    my $message = shift;
    if ($debug) {
        print "$message\n";
    }
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07

cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
#      e-Monitoring Networks, Inc.  All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
    my $class = shift;
    my $self = {};

    $self->{'debug'} = 0;
    bless $self, $class;

    return $self;
}

sub connect {
    my $self = shift;
    my $dbname = shift;
    my $dbhost = shift;
    my $dbuser = shift;
    my $dbpass = shift;

    my $host = '';
    if (defined($dbhost)) {
        $host = ";host=$dbhost";
    }

    eval {
        $self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
        $self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
    };
    if ($self->{'dbh'}) {
        return 1;
    }
    $self->{'error'} = "Error connecting to database $@";
    $self->debug("Error connecting to database $@");
    return 0;
}

sub query {
    my $self = shift;
    my $query = shift;

    $self->debug($query, 1);
    my $sth;
    eval {
        $sth = $self->{'dbh'}->prepare($query);
    };
    unless ($sth) {
        $self->{'error'} = "error preparing query $@";
        $self->debug("error preparing query $@");
        return undef;
    }
    my $qty;
    eval {
        $qty = $sth->execute;
    };
    unless ($qty) {
        $self->{'error'} = "error executing query $@";
        warn "error executing query $@ $query";
        return undef;
    }
    $self->debug("returning $qty, $sth from query", 6);
    return ($qty, $sth);
}

sub disconnect {
    my $self = shift;

    $self->{'dbh'}->disconnect;
    return 0;
}

sub debug {
    my $self = shift;
    my $msg = shift;
    my $level = shift || 0;

    if ($level < $self->{'debug'}) {
        print "$msg\n";
    }
    return 0;
}
1;

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

setup traffic db[edit]

Install mysql:

cd /usr/ports/databases/mysql50-server
make install clean

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:

/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:

vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

Install mysql perl database modules:

cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)

Setting up database

rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

If this was a new server we'd setup new tables. See mysql for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):

rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

process flows from bwdb2[edit]

On traffic database server (bwdb):

crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl

Add access to mysql:

mysql -u root -p 
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';

cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
    if (-e "$sqldir/.lock") {
        open(FILE, "$sqldir/.lock");
        my $pid = <FILE>;
        chomp($pid);
        close(FILE);
        if (kill(0, $pid)) {
            #another process is using the queue, bail out
            exit(0);
        }
        else {
            #dead lock file, remove it
            `rm $sqldir/.lock`;
        }
    }
    open(FILE, "> $sqldir/.lock");
    print FILE "$$\n";
    close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
   next unless $file =~ /done$/;
   my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
   unless ($?==0) {
      push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
   }
   else {
      `rm $sqldir/$file`;
   }
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
   email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
    my $subj=shift;
    my $body=shift;
    use Mail::Sendmail;

    # prepare message
    my %mail = (
        To      => 'dave@johncompanies.com',
        From    => 'support@johncompanies.com',
        Subject => $subj,
        Message => $body,
        smtp    => 'mail.johncompanies.com',
    );
    sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
    my $message = shift;
    if ($debug) {
        print "$message\n";
    }
}

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to /etc/ipnat.rules

# bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32

Reload:

ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:

ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:

cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me 
ipfw add 5 allow ip from 69.55.230.2 25 to me 
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:

cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:

ssh 69.55.233.199 hostname

bwdb2[edit]

Summary[edit]

This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

Location: i2b, cab6
OS: FreeBSD 6.4 x86
Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
Hardware: Custom 2U. Single power supply.
Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

Services Provided[edit]

netflow
bigbrother

netflow[edit]

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under /usr/home/flows/ (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)

1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue: /usr/home/working

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:

2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:

8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

Regular maintenance[edit]

Check RAID array

if space becomes tight, move sql files and flow files to backup server, both located in /usr/home/flowbin/archive

firewall (newgateway)[edit]

Summary[edit]

This machine is the primary (only) firewall for the entire network at castle.

Location: castle, cab 3-8
OS: FreeBSD 4.11 x86
Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

Services Provided[edit]

firewall (ipfw)
snmp
bigbrother

Firewall Rule Configuration[edit]

See Firewall Rule Configuration for more discussion on how to actually manipulate firewall rules.

Disaster Recovery[edit]

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"


#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248

Cronjobs[edit]

1 0 * * * /usr/local/etc/rsync.backup

Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3  4 5 17331

Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside /etc/daily.local you will see a call to /etc/makepiperules.pl This script will create /etc/firewall.sh which contains all the firewall and pipe rules in place at the time the script was run.

DOS attacks[edit]

See Handling a DoS attack regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will

enter a rule that allows all UDP to go through
send an emergency email to support and indicating an attack is in progress
send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1

This file lives under /usr/home/user/doswatch.pl To run:

cd /usr/home/user
./doswatch.pl &

To kill;

fg
^C

It writes its findings to /usr/home/user/doswatch.log

backup1[edit]

Summary[edit]

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

Location: castle, cab 3-8
OS: Ubuntu 8.04.1 server x86
Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

Services provided[edit]

backup via rsync
mysql - traffic data
nfs server - for backups
snmp client - for big brother
bigbrother client

Usage and Notes[edit]

all data is stored under /data
virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: /backup1), and they all have their ssh keys setup to allow passwordless rsync's
each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
cancelled customer systems are compressed and stored under /data/deprecated
archived bwdb2 flow files are stored under /data/bwdb2
critical files from backup2 are stored under /data/backup2

Cronjobs[edit]

00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate  /data/backuplog.log

this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

10,25,40,55 * * * * /usr/local/sbin/processsql.pl

this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.

0 0 * * * /usr/local/sbin/3wraidchk

checks the health of the RAID array

Regular maintenance[edit]

build[edit]

Setup raid5 with a boot vol of 12G 5.45tb 
12G boot
4664 GB 

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number  Start  End  Size  File system  Name  Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number  Start   End     Size    File system  Name     Flags
# 1      17.4kB  4987GB  4987GB               primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
#        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
#        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
#        102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables:   967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem            Size  Used Avail Use% Mounted on
#/dev/sda2             8.3G  540M  7.3G   7% /
#varrun               1013M   40K 1013M   1% /var/run
#varlock              1013M     0 1013M   0% /var/lock
#udev                 1013M   56K 1013M   1% /dev
#devshm               1013M     0 1013M   0% /dev/shm
#/dev/sdb1             4.5T  192M  4.3T   1% /data


apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3        backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty 
# 
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0 


vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines: 
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity  jcread 10.1.4.5
rocommunity  jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface


echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab


vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh 
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT 
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"


echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart


echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd


echo '10.1.4.8                backup1' >> /etc/hosts
echo '/dev/sdb1	/data  ext3  rw,noatime  0  0' >> /etc/fstab


to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server 
apt-get install make
apt-get install gcc
apt-get install g++	
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
  /data/mysql/ r,
  /data/mysql/** rwk,
Comment out:
#  /var/lib/mysql/ r,
#  /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on


0 0 * * * /usr/local/sbin/3wraidchk

backup2[edit]

Summary[edit]

THIS SERVER IS OUT OF SERVICE AND REPLACED BY BACKUP4

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

Location: castle, cab 3-7
OS: FreeBSD 6.1 x86
Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
Drives:
- 3ware 7500-8:
  - 200 GB JBOD (1 x 200G) labeled 0-0
  - 500 GB RAID5 (3 x 250G) 0-1 thru 0-3
  - 700 GB RAID5 (4 x 250G) 0-4 thru 0-7
- 3ware 7500-8:
  - 700 GB RAID5 (4 x 250G) 1-0 thru 1-3
  - 700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:

ifconfig fxp0 link 00:90:27:f9:0a:d9

Services provided[edit]

backup via rsync and nfs
samba
nfs
snmp
bigbrother

Usage[edit]

all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: /mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4
iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /mnt/data2/iso
this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
this server serves as an archive for exported db data from bwdb and old flow files.
isys backs up here
customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
in /mnt/data4 there are lots of useful things used for building our vps servers, customer servers, and management scripts:
- /bin: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
- /build: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
- /vzrpms: contains the OS templates for many-to-most of the OS's we offer on vz systems

Cronjobs[edit]

backs itself up nightly to nfs-mounted backup1 (mountpoint: /backup2)

Regular maintenance[edit]

Check on health

backup3[edit]

Summary[edit]

This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

Location: i2b, cab 6
OS: Ubuntu 10.04.1 server amd64
Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

Services provided[edit]

backup via rsync and nfs
samba
nfs
digi realport
snmp
bigbrother

Usage[edit]

all data is stored under /data
iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /data/iso
this server serves as an archive for exported db data from bwdb and old flow files.
inftrastructure machines at i2b back up here
customers are nfs-moutned under /data/customers as file-backed loopback devices

management scripts[edit]

mkbackups

mkbackup <cid> GB <ip>

Cronjobs[edit]

0 0 * * * /usr/local/sbin/arecaraidchk

RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive

Rotate daily snapshots for infrastructure machine backups

Regular maintenance[edit]

Check on RAID health

Build[edit]

BIOS Config[edit]

disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

Install OS[edit]

Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba

DNS and private IP[edit]

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net

vi /etc/network/interfaces

auto eth0
iface eth0 inet static
        address 69.55.229.4
        netmask 255.255.255.0
        network 69.55.229.0
        broadcast 69.55.229.255
        gateway 69.55.229.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 69.55.229.3 66.181.0.2
        dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
        address 69.55.231.2
        netmask 255.255.255.0
        network 69.55.231.0
        broadcast 69.55.231.255

auto eth1
iface eth1 inet static
        address 10.1.2.3
        netmask 255.255.255.0
        network 10.1.2.0
        broadcast 10.1.2.255

Install packages[edit]

apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server

tweak grub, enable serial[edit]

vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub

echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf

install realport (digi) driver[edit]

give the digi an ip with DgIpServ.exe

cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz 
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults

configure ports:

dgrp_cfg_node init el 10.1.2.10 16

try connecting with:

cu -l /dev/ttyel00 -s 38400

shell, ntp, ssh key, hosts[edit]

Shell autocompletion search:

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

Setup ntp:

vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com

Generate ssh keys:

cd /root/
ssh-keygen -t dsa

Defaults, no password

Setup hosts:

echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

Copy keys to servers where we need passwordless login:

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

Setup shell:

vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
 source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity  jcread 10.1.4.5
rocommunity  jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
 echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
 /etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
 echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
 echo "bb:x:1984:" >> /etc/group
 pwconv

Create home:
 mkdir /home/bb
 chown bb.bb /home/bb
 cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc

Configure main bb server:

echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh

(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh

(y to questions)

./bbchkhosts.sh

(ignore ssh errors)

cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

make; make install
cd ..
./runbb.sh start
more BBOUT

(look for errors)

exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:

ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199

raid check[edit]

3ware[edit]

scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk

areca[edit]

cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info

scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk

cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1

misc binaries[edit]

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive

(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];


my %mail = (
   To      => $to,
   From    => $to,
   Subject => $sub,
   Message => '',
   smtp    => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

chmod 0700 /bin/mail

mkbackup[edit]

mkdir /data/customers

cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
  cid=$1
else
  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
  exit
fi

if test $2; then
  gb=$2
else
  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
  exit
fi

if test $3; then
  ip=$3
else
  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
  exit
fi


if test -e /data/customers/${cid}-file; then
  echo "ERROR: /data/customers/${cid}-file exists"
  exit
else
  echo "touch /data/customers/${cid}-file"
  touch /data/customers/${cid}-file
  count=`echo $gb|awk '{print $1*1000}'`
  echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
  dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
  echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
  /sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
  echo "ERROR: /data/customers/$cid exists"
  exit
else
  echo "mkdir /data/customers/${cid}"
  mkdir /data/customers/${cid}
  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
  mount -o loop /data/customers/${cid}-file /data/customers/$cid
  df -h /data/customers/$cid

  echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
  echo "" >> /etc/nfs_backup_mounts.sh

  echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
  /etc/init.d/nfs-kernel-server restart
  tail /var/log/messages
fi

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local

add:

/etc/nfs_backup_mounts.sh

samba[edit]

apt-get install samba

vi /etc/samba/smb.conf

comment out any mounts, add

[data]
   read only = yes
   locking = no
   path = /data/iso
   guest ok = yes

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .

Moving from one server to another[edit]

Here are the steps you would take to move settings and data from one server to a new backup server:

rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)

rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/

after umounting all the customers, copy over the (empty) directories separately:

for f in `find .  -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

copy mount script

[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

copy rc.local

[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

copy /etc/exports

[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

edit /etc/hostname on both machines (set current to oldbackup3)

edit /etc/network/interfaces (swap IPs).

stop mounts from mounting on old and new servers so it doesnt start with reboot right away:

chmod 000 /etc/nfs_backup_mounts.sh

reboot both servers @ same time

check everything out

run /etc/nfs_backup_mounts.sh on new server

if switch port changed update mrtg to reflect correct port pub nic is on (on p20):

vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

backup4[edit]

Summary[edit]

This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs. Only FreeBSD virt customers directly accesses this server to perform their backups.

Location: castle, cab 3-7
OS: FreeNAS 9.3 (FreeBSD 9.3)
Networking: Priv IP: 10.1.2.9/24 AND 10.1.7.9/24, Pub IPs: 69.55.230.6/24
Hardware: JC-08014

           Intel S5000VSA Motherboard
           1 x Intel Xeon E5410  @ 2.33GHz CPU
           3ware 9690SA-8I RAID Card w BBU
           16GB RAM
           Dual power supply.

Drives: 7 TB (6 x 2TB) ZFS RAIDZ2 array running on JBOD

      1 128 GB SSD system drive and 6 drive SATA bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap.

GUI management at http://backup4.johncompanies.com

Services provided[edit]

backup via rsync and nfs
samba
nfs
snmp?
bigbrother?

Usage[edit]

all data is stored under /data
iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /data/iso ??
this server serves as an archive for exported db data from bwdb and old flow files. ??
customers are nfs-moutned under /data/users (/mnt/zfs/users) as zfs ?

management scripts[edit]

mkbackups?

mkbackup <cid> GB <ip>

Cronjobs[edit]

0 0 * * * /usr/local/sbin/arecaraidchk

RAID checks ?

35 4 * * * /usr/local/sbin/snapshot_archive

Rotate daily snapshots for infrastructure machine backups

00 15 * * * /usr/local/sbin/snapshot_rotate

Rotate daily snapshots for customer machine backups

Regular maintenance[edit]

Check on RAID health

Build[edit]

console[edit]

Summary[edit]

This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

OBSOLETE

Location: castle, cab 3-8
OS: SunOS 5.8 (solaris)
Networking: Priv IP: 10.1.4.4
Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the tip command to connect to devices listed in /etc/remote

SSH WORK ~2021 ssh user@console.johncompanies.com

PW: 674*****

i.e.

tip switch-p1
tip jail1

Configuring digi/ports[edit]

/etc/remote[edit]

This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. jail4) and the speed (i.e. 9600). You can decipher which port on the digi each line corresponds to by the CP001s or CO001s (port 1 on digi1 and digi2), CP002s or CO002s (port 2 on digi1 and digi2)

drpadmin[edit]

The tool you use to configure a device to a digi box is drpadmin:

bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0       10.1.4.10       32      CO      771     never   1027
1       10.1.4.11       32      CP      771     never   1027
2       65.116.11.2     8       el      771     never   1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :

Use those commands above to modify the devices available.

Switching IP/hostname[edit]

Edit:

/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

devweb[edit]

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:

httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our main site.

firewall2[edit]

Summary[edit]

This machine is the primary firewall for the entire network at i2b. firewall3 is a hot standby replacement for firewall2. Both firewall2 and firewall3 should not be connected at the same time since they use the same internal and external IP addresses.

Location: i2b, cab 6
OS: FreeBSD 6.4 x86
Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

Services Provided[edit]

firewall (ipfw)
bigbrother for customer machines

Firewall Rule Configuration[edit]

See Firewall Rule Configuration for more discussion on how to actually manipulate firewall rules.

Disaster Recovery[edit]

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3  netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1  netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1  netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1  netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1  netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1  netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1  netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1  netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1  netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

Cronjobs[edit]

30 3 * * * /usr/local/etc/rsync.backup Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5

Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count

Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;

This script will create /etc/firewall.sh which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk

Checking the health of the RAID array

DOS attacks[edit]

See Handling a DoS attack regarding how to handle a DOS attack.

build[edit]

partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf 
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf


6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0   "/usr/libexec/getty std.9600"   vt100   on  secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console


7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024 
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' 

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname


10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel. 

cd /usr/src/sys/i386/conf 
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld 
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3  netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="firewall2,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
        $1 $TOPARGS > $BBTMP/TOP.$$
#        /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2


27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
 (edit the PTR too)


33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to 
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup 
vi /usr/local/etc/rsync.backup 
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup


34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
    if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; 
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries
	
scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139

firewall3[edit]

Summary[edit]

This machine is the backup firewall for the network at i2b.

Location: i2b, cab ?
OS: FreeBSD 9.1 amd64
Networking: Priv IP: 10.1.2.5, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks.


The internal network NIC is the left one on the motherboard (69.55.229.1/24, ...).
The external network NIC is the right one on the motherboard (66.181.18.3/28).
The PCI ethernet card is connected to our private network (10.1.2.5/24).

Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
Drives: 160 GB (2 x 160GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

Services Provided[edit]

firewall (ipfw)
bigbrother

Firewall Rule Configuration[edit]

See Firewall Rule Configuration for more discussion on how to actually manipulate firewall rules.

Disaster Recovery[edit]

To put the backup firewall3 into service:

Move the internal cable (to our networks) from firewall2 to em1 which is the left most ethernet port (69.55.229.1).
Move the external cable (to outside world) from firewall2 to em0 which is the port to the right on the motherboard (66.181.18.3).
The PCI ethernet port (fxp0) should already be connected to private network (10.1.2.5).

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall3.johncompanies.com"
ifconfig_em0="inet 66.181.18.3  netmask 255.255.255.224"

ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"

# ifconfig_em1_alias1="inet 65.50.228.1  netmask 255.255.255.0"
# ifconfig_em1_alias2="inet 65.50.229.1  netmask 255.255.255.0"
# ifconfig_em1_alias3="inet 65.50.230.1  netmask 255.255.255.0"
# ifconfig_em1_alias4="inet 65.50.231.1  netmask 255.255.255.0"
# ifconfig_em1_alias5="inet 65.50.232.1  netmask 255.255.255.0"
# ifconfig_em1_alias6="inet 65.50.233.1  netmask 255.255.255.0"
# ifconfig_em1_alias7="inet 65.50.234.1  netmask 255.255.255.0"
# ifconfig_em1_alias8="inet 65.50.235.1  netmask 255.255.255.0"

ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

sshd_enable="YES"
usbd_enable="YES"

Cronjobs[edit]

30 3 * * * /usr/local/etc/rsync.backup Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5

Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count

Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;

This script will create /etc/firewall.sh which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk

Checking the health of the RAID array

DOS attacks[edit]

See Handling a DoS attack regarding how to handle a DOS attack.

build[edit]

partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf 
echo "WITHOUT_X11=yes \
KERNCONF=firewall3 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf


6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0   "/usr/libexec/getty std.9600"   vt100   on  secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console


7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024 
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys' 

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname


10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la        ls -aG
alias lf        ls -FAG
alias ll        ls -lAG
alias ls        ls -AG
alias mbm       mb mount
alias mbu       mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui 
make install clean; rehash; mail -s 'cvs installed' 8583619553@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src 
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' 8583619553@vtext.com < /dev/null

13. configure new kernel. 

cd /usr/src/sys/amd64/conf 
scp backup2:/mnt/data4/build/freebsd/firewall3-9.1 ./firewall3

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' 8583619553@vtext.com < /dev/null
(supermicro: 2:15 mins, 2950: 38? mins)
make installworld 
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"

fsck_y_enable="YES"
background_fsck="NO"

hostname="firewall3.johncompanies.com"
#  external network
ifconfig_em0="inet 66.181.18.3  netmask 255.255.255.224"

#  internal network
ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_em1_alias0="inet 69.55.231.1  netmask 255.255.255.0"

ifconfig_em1_alias1="inet 65.50.228.1  netmask 255.255.255.0"
ifconfig_em1_alias2="inet 65.50.229.1  netmask 255.255.255.0"
ifconfig_em1_alias3="inet 65.50.230.1  netmask 255.255.255.0"
ifconfig_em1_alias4="inet 65.50.231.1  netmask 255.255.255.0"
ifconfig_em1_alias5="inet 65.50.232.1  netmask 255.255.255.0"
ifconfig_em1_alias6="inet 65.50.233.1  netmask 255.255.255.0"
ifconfig_em1_alias7="inet 65.50.234.1  netmask 255.255.255.0"
ifconfig_em1_alias8="inet 65.50.235.1  netmask 255.255.255.0"

defaultrouter="66.181.18.2"

#  private network
ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.5"

sshd_enable="YES"
usbd_enable="YES"
ntpd_enable="YES"
# powerd_enable="YES"


20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports

echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_9_1\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' 8583619553@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
cd /usr/local/sbin 
tar xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username   : bb
Password   : <random>
Full Name  : bb
Uid        : 1984
Class      :
Groups     : bb
Home       : /home/bb
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.5 firewall3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles 
MACHINE="firewall3,johncompanies,com"      # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh 
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
        $1 $TOPARGS > $BBTMP/TOP.$$
#        /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT 
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2


27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall3.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
 (edit the PTR too)


33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to 
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup 
vi /usr/local/etc/rsync.backup 
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup


34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.5

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
    if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl; 
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries
	
scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139

wiki[edit]

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

Setup jail[edit]

mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset 

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf 

vi /etc/crontab
 
(remove the adjkerntz lines )
 
vi /etc/periodic/security/100.chksetuid
 
replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
 with: MP='/' (use single quotes) 

mkdir -p /usr/compat/linux/dev
 
adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username   : user
Password   : <random>
Full Name  : user
Uid        : 1001
Class      :
Groups     : user
Home       : /home/user
Home Mode  :
Shell      : /bin/sh
Locked     : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye! 

vi /usr/home/user/.profile 
TERM=vt100;     export TERM
 
tzsetup
 
newaliases 
 
rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot
 
vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit               /dev/console
*.err;kern.warning;auth.notice;mail.crit                /var/log/messages 

exit
exit
 
cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1
 
cp -r /usr/ports /mnt/data1/wiki-dir/usr 

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh

mediawiki setup[edit]


cd /usr/ports/net/rsync
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
    <FilesMatch "\.ph(p3?|tml)$">
        SetHandler application/x-httpd-php
    </FilesMatch>
    <FilesMatch "\.phps$">
        SetHandler application/x-httpd-php-source
    </FilesMatch>
    # To re-enable php in user directories comment the following lines
    # (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
    # prevents .htaccess files from disabling it.
    <IfModule mod_userdir.c>
        <Directory /home/*/public_html>
            php_admin_value engine Off
        </Directory>
    </IfModule>
</IfModule>


cat > /usr/local/etc/apache22/extra/vhost-wiki.conf 
<VirtualHost *:443>
        ServerAdmin support@johncompanies.com

        DocumentRoot /usr/local/www/wiki
#        <Directory />
#                Options FollowSymLinks
#                AllowOverride None
#                Order deny,allow
#        </Directory>
        <Directory /usr/local/www/wiki>
                Options Indexes FollowSymLinks MultiViews
                Deny from all
                AllowOverride AuthConfig
                Order allow,deny
                DirectoryIndex index.php
                #Allow from 69.55.233.195
                #Allow from boody.dyndns.org
        </Directory>

        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        <Directory "/usr/lib/cgi-bin">
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        </Directory>

        ErrorLog /var/log/httpd-error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog /var/log/httpd-access.log combined

    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>

    SSLEngine on
    SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
    SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com

cat > /usr/local/www/wiki/LocalSettings.php

<?php
# This file was automatically generated by the MediaWiki 1.19.2
# installer. If you make manual changes, please keep track in case you
# need to recreate them later.
#
# See includes/DefaultSettings.php for all configurable settings
# and their default values, but don't forget to make changes in _this_
# file, not there.
#
# Further documentation for configuration settings may be found at:
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# Protect against web entry
if ( !defined( 'MEDIAWIKI' ) ) {
	exit;
}

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

$wgSitename      = "JCWiki";

## The URL base path to the directory containing the wiki;
## defaults for all runtime URL paths are based off of this.
## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
$wgScriptPath       = "";
$wgScriptExtension  = ".php";

## The protocol and server name to use in fully-qualified URLs
$wgServer           = "https://69.55.229.8";

## The relative URL path to the skins directory
$wgStylePath        = "$wgScriptPath/skins";

## The relative URL path to the logo.  Make sure you change this from the default,
## or else you'll overwrite your logo when you upgrade!
#$wgLogo             = "$wgStylePath/common/images/wiki.png";
$wgLogo             = "$wgStylePath/common/images/jclogo.gif";

## UPO means: this is also a user preference option

$wgEnableEmail      = true;
$wgEnableUserEmail  = true; # UPO

$wgEmergencyContact = "apache@69.55.229.8";
$wgPasswordSender   = "apache@69.55.229.8";

$wgEnotifUserTalk      = false; # UPO
$wgEnotifWatchlist     = false; # UPO
$wgEmailAuthentication = true;

## Database settings
$wgDBtype           = "mysql";
$wgDBserver         = "localhost";
$wgDBname           = "my_wiki";
$wgDBuser           = "root";
$wgDBpassword       = "";

# MySQL specific settings
$wgDBprefix         = "";

# MySQL table options to use during installation or update
$wgDBTableOptions   = "ENGINE=InnoDB, DEFAULT CHARSET=binary";

# Experimental charset support for MySQL 5.0.
$wgDBmysql5 = false;

## Shared memory settings
$wgMainCacheType    = CACHE_NONE;
$wgMemCachedServers = array();

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads  = false;
#$wgUseImageMagick = true;
#$wgImageMagickConvertCommand = "/usr/bin/convert";

# InstantCommons allows wiki to use images from http://commons.wikimedia.org
$wgUseInstantCommons  = false;

## If you use ImageMagick (or any other shell command) on a
## Linux server, this will need to be set to the name of an
## available UTF-8 locale
$wgShellLocale = "en_US.utf8";

## If you want to use image uploads under safe mode,
## create the directories images/archive, images/thumb and
## images/temp, and make them all writable. Then uncomment
## this, if it's not already uncommented:
#$wgHashedUploadDirectory = false;

## Set $wgCacheDirectory to a writable directory on the web server
## to make your wiki go slightly faster. The directory should not
## be publically accessible from the web.
#$wgCacheDirectory = "$IP/cache";

# Site language code, should be one of the list in ./languages/Names.php
$wgLanguageCode = "en";

$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";

# Site upgrade key. Must be set to a string (default provided) to turn on the
# web installer while LocalSettings.php is in place
$wgUpgradeKey = "3196710f4a7d7332";

## Default skin: you can change the default skin. Use the internal symbolic
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
$wgDefaultSkin = "vector";

## For attaching licensing metadata to pages, and displaying an
## appropriate copyright notice / icon. GNU Free Documentation
## License and Creative Commons licenses are supported so far.
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
$wgRightsUrl  = "";
$wgRightsText = "";
$wgRightsIcon = "";

# Path to the GNU diff3 utility. Used for conflict resolution.
$wgDiff3 = "/usr/bin/diff3";

# Query string length limit for ResourceLoader. You should only set this if
# your web server has a query string length limit (then set it to that limit),
# or if you have suhosin.get.max_value_length set in php.ini (then set it to
# that value)
$wgResourceLoaderMaxQueryLength = -1;



# End of automatically generated settings.
# Add more configuration options below.

copy/backup wiki[edit]

on main/primary wiki:

/usr/local/etc/rc.d/mysql-server stop
ssh 69.55.230.18 "/etc/init.d/mysql stop"
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/
/usr/local/etc/rc.d/mysql-server start
ssh 69.55.230.18 "/etc/init.d/mysql start"

@@ Line 3: / Line 3: @@
 == jail1 ==
-* Location: castle, cab 3-
+* Location: castle, SHUTDOWN
-* OS: FreeBSD  x86
+* OS: FreeBSD 6.2 i386
-* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
+* Networking: Priv IP: 10.1.4.101 (PCI nic), Pub IP: 69.55.230.107 (onboard)
 * Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
-* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Remote management: none
+* Disk accounting: gvinum
 == jail2 ==
-* Location: castle, cab 3-
+* Location: castle, cab 6-16
-* OS: FreeBSD  x86
+* OS: FreeBSD 7.2 amd64
-* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
+* Networking: Priv IP: 10.1.4.102, Pub IP: 69.55.228.53 (2 onboard nics)
 * Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
-* Drives: two  GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.
+* Drives: one 146 GB (2 x 146GB) RAID1 array, two 300 GB (4 x 300GB) RAID1 arrays running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.232
+* Disk accounting: md
+== jail3 ==
+* Location: I2b  SHUTDOWN
+* OS: FreeBSD 8.3 amd64
+* Networking: Priv IP: 10.1.2.103, Pub IP: 69.55.229.7 (2 onboard nics)
+* Hardware: Supermicro (custom build). 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: two 300 GB (2 x 300GB) RAID1 array running on a 3ware 8006-2LP RAID card.
+* Remote management: none
+* Disk accounting: md
+=== Notes ===
+* '''We should not add users to this server since it is at I2B'''
+* must be ssh'd to from nat2
+* is a super jail for customer col01737
+== jail4 ==
+* Location: castle, cab 6-17
+* OS: FreeBSD 9.1 x86_64
+* Networking: Priv IP: 10.1.4.104, Pub IP: 69.55.228.104 (2 onboard nics)
+* Hardware: Dell 2850. 6 x 300GB SCSI drives (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* CPU:  2 x Intel(R) Xeon(TM) CPU 2.80GHz (8 virtual CPUs)
+* RAM:  16 GB ( 4 x 4GB Reg ECC )
+* Drives: one 1.4 TB RAID 5 array (6 x 300GB SCSI) Dell-branded (PERC 4e)LSI megarc RAID card.
+* Remote management: None
+* Disk accounting: md
+=== Notes ===
+Only FreeBSD 9.1 jail
+Not upgraded to FBSD 9.2 or 9.3 because too many libraries modified (would require customers to rebuild apps).
+== jail5 ==
+* Location: castle, cab 3-6
+* OS: FreeBSD 10.1 x86_64
+* Networking: Priv IP: 10.1.4.105, Pub IP: 69.55.230.105 (2 onboard nics)
+* Hardware: Supermicro JC-14004 - Intel S1200BTL motherboard - 6 SATA/SAS drive bays (2 colums of 3), Dual power supply.
+* CPU:  1 x Intel(R) Xeon(TM) E3-1230 V2 CPU 3.30 GHz (8 virtual CPUs)
+* RAM:  32 GB ( 4 x 8GB ECC )
+* Drives: 1x80 GB SATA SSD on motherboard + one 2.6 TB RAID 5 array 4x1 TB + 3ware 9650 RAID card.
+* Remote management: Intel RMM 4 - 10.1.4.235
+* Disk accounting: md
+=== Notes ===
+Only FreeBSD 10.1 jail used for bhyve virtuals.
+Use ~+Ctrl-D to disconnect from console (vm attach colXXXXX).
+== jail6 ==
+* Location: castle, cab 6-16
+* OS: FreeBSD 10.3 x86_64
+* Networking: Priv IP: 10.1.4.106, Pub IP: 69.55.230.106 (2 onboard nics)
+* Hardware: Supermicro JC-14004 - Intel S1200BTL motherboard - 6 SATA/SAS drive bays (2 colums of 3), Dual power supply.
+* CPU:  1 x Intel(R) Xeon(TM) E3-1230 V2 CPU 3.30 GHz (8 virtual CPUs)
+* RAM:  32 GB ( 4 x 8GB ECC )
+* Drives: one 2.7 TB ZFS RAID 6 array 5x1 TB
+* Remote management: Intel RMM 4 - 10.1.4.236
+* Disk accounting: zfs
-* Location: castle, cab 3-
+== jail7 ==
-* OS: FreeBSD  x86
-* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
+* Location: castle, cab 3-5  SCHEUDLED SHUTDOWN 9/30/19
+* OS: FreeBSD 6.3 i386
+* Networking: Priv IP: 10.1.4.107, Pub IP: 69.55.230.108 (2 onboard nics)
+* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: two 146 GB (4 x 146GB) RAID1 arrays, one 74 GB (2 x 74GB) RAID1 array running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.237
+* Disk accounting: gvinum
+=== Notes ===
+Do not run a verify while OS/jails running, will crash.
+== jail8 ==
+* Location: castle, cab 3-6
+* OS: FreeBSD 8.0 amd64
+* Networking: Priv IP: 10.1.4.108, Pub IP: 69.55.234.2 (2 onboard nics)
+* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (PERC 6/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.238
+* Disk accounting: md
+== jail9 ==
+* Location: castle, cab 3-6
+* OS: FreeBSD 8.2 amd64
+* Networking: Priv IP: 10.1.4.109, Pub IP: 69.55.232.36 (2 onboard nics)
+* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 300GB) RAID1 array running on an LSI-based, Dell-branded (PERC 5/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.239
+* Disk accounting: md
+== jail11 ==
+* Location: castle, cab 3-7
+* OS: FreeBSD 4.7 i386
+* Networking: Priv IP: 10.1.4.111 (PCI nic), Pub IP: 69.55.236.92 (onboard)
+* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Remote management: none
+* Disk accounting: vinum
+== mx1 ==
+* Location: castle, SHUTDOWN AND SCRAPPED
+* OS: FreeBSD 4.11 i386
+* Networking: Priv IP: 10.1.4.201 (PCI nic), Pub IP: 69.55.237.3 (onboard)
 * Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
-* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Remote management: none
+* Disk accounting: vinum
+=== Notes ===
+* is our (old) backup mail/dns vps service host
+== mx2 ==
+* Location: castle, SHUTDOWN AND SCRAPPED
+* OS: FreeBSD 7.1 i386
+* Networking: Priv IP: 10.1.4.202 (PCI nic), Pub IP: 69.55.237.90 (onboard)
+* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: two 74 GB (4 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Disk accounting: gvinum
+=== Notes ===
+* is our latest backup mail/dns vps service host
+== jail17 ==
+* Location: castle, cab 3-7
+* OS: FreeBSD 4.10 i386
+* Networking: Priv IP: 10.1.4.117 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
+* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S  RAID card.
+* Remote management: none
+* Disk accounting: vinum
+* Host of devweb.johncompanies.com and www.utopian.com/mail.utopian.com
+== jail18 ==
+* Location: castle, cab 3-5  SCHEDULED SHUTDOWN 9/30/19
+* OS: FreeBSD 4.10 i386
+* Networking: Priv IP: 10.1.4.118 (PCI nic), Pub IP: 69.55.228.2 (onboard nics)
+* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: two 74 GB (4 x 74GB) RAID1 arrays, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S  RAID card.
+* Remote management: none
+* Disk accounting: vinum
+* Host of ns2c.johncompanies.com (now on ns2c.johncompanies.com on ganeti)
+== jail19 ==
+* Location: castle, cab 3-5  SCHEDULED SHUTDOWN 9/30/19
+* OS: FreeBSD 6.1 i386
+* Networking: Priv IP: 10.1.4.119 (PCI nic), Pub IP: 69.55.228.200 (onboard nics)
+* Hardware: Supermicro (custom build). 6 SCA SCSI drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an Adaptec 2120S  RAID card.
+* Remote management: none
+* Disk accounting: gvinum
 = virts =
@@ Line 30: / Line 185: @@
 == quar1 ==
-* Location: castle, cab 3-
+* Location: castle, SHUTDOWN AND SCRAPPED
-* OS: FreeBSD  x86
+* OS: RedHat 7.3 x86
-* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
+* Networking: Priv IP: 10.1.4.151 (PCI nic), Pub IP: 69.55.227.2 (onboard nic)
 * Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
-* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Drives: one 36 GB (2 x 36GB) RAID1 array, one 74 GB (2 x 74GB) RAID1 array, running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Remote management: none
+* Virtuozzo version: 2.6.1
+* VZ license: hwid=23C0.C0E1.6FDD.08BA.8971.8E1C.EBD5.1EDC serial=0DE6.903E.E239.E23F.470C.4369.4104.A5A4
+=== Notes ===
+* used to be the home of customers who's VE's would just run out of control/badly
+* has a max of 10 VE's allowed to run
 == virt9 ==
-* Location: castle, cab 3-
+* Location: castle, cab 3-7
-* OS: FreeBSD  x86
+* OS: RedHat 7.3 x86
-* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
+* Networking: Priv IP: 10.1.4.59 (PCI nic), Pub IP: 69.55.226.161 (onboard nic)
 * Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
-* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Drives: two 74 GB (2 x 74GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.
+* Remote management: none
+* Virtuozzo version: 2.6.1
+* VZ license: hwid=BC15.B4D6.0D25.A5FE.F3BA.D518.E351.AE3F serial=F6AD.B6B4.5650.8869.C97C.73EE.AF65.FA8B
 == virt11 ==
-* Location: castle, cab 3-
+* Location: castle, cab 3-6
-* OS: CentOS
+* OS: CentOS 5.4 x86
-* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
+* Networking: Priv IP: 10.1.4.61, Pub IP: 69.55.238.3, 2 onboard nics
 * Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
-* Drives: two  GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.
+* Drives: one 146 GB (2 x 146GB) RAID1 array, one 400 GB (2 x 400GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.211
+* Virtuozzo version: 4.0.0
+* VZ license: hwid="029D.A187.78E1.480F.49E3.E20A.7389.7F79" serial="163C.F3E2.195F.96B5.2D38.8937.9600.4A05"  key_number="VZ.00172378.0006"
+  vzlicload -p A40R00-D8CS00-5D8817-A2RB23-8Y9J78
+== virt12 ==
-== virt15 ==
+* Location: castle, cab 3-7
+* OS: CentOS 5.2 x86
+* Networking: Priv IP: 10.1.4.62, Pub IP: 69.55.227.70, 2 onboard nics
+* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: two 300 GB (2 x 300GB) RAID1 arrays one 400 GB (2 x 400GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.212
+* Virtuozzo version: 4.0.0
+* VZ license: hwid="0C53.A413.E095.B4F4.51BC.D740.6919.A77B" serial="84E5.9498.3759.E683.E24B.2514.CA72.DC31"
-* Location: castle, cab 3-
+== virt13 ==
-* OS: CentOS
-* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
+* Location: castle, cab 6-17
-* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Switch port:  P13-
-* Drives: two  GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.
+* OS: CentOS 6.2 x86_64
+* Networking: Priv IP: 10.1.4.63, Pub IP: 69.55.226.2, 2 onboard nics
+* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* CPU:    2 x Intel(R) Xeon(R) CPU E5420  @ 2.50GHz  (8 virtual cores)
+* RAM:    32 GB (8 x 4GB DDR2 FB-DIMM ECC 667MHz)
+* Drives: one 146 GB (2 x 146GB) RAID1 array, one 600 GB (2 x 600GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI-based, Dell-branded (perc 6/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.213
+* Virtuozzo version: 4.7.0
+* VZ license: hwid="7D07.93BE.0B1F.7D2B.B039.4B5B.48B6.453B" serial="60A4.A94C.44BB.DCD6.8D03.1778.605B.10FE"
+=== Notes ===
+* home to our latest/current signups
+* currently the only 64bit vz host
+== virt14 ==
+* Location: castle, cab 6-16
+* Switch Port: p13-
+* OS: CentOS 6.4 x86_64
+* Networking: Priv IP: 10.1.4.64 Pub IP: 69.55.225.14 2 onboard nics
+* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* CPU: 2 x Xeon 5140 Dual Core @ 2.33GHz (4 virtual CPUs)
+* RAM: 32 GB  (8 x 4GB Reg ECC)
+* Drives: one 146 GB (2 x 146 GB SAS) RAID1 array, and one 1TB RAID1 array (2 x 1 TB SATA), running on an LSI-based, Dell-branded (perc 5/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.214
+* Virtuozzo version: 4.7.0
+* VZ license: hwid="EA32.2CA0.2368.F5FC.DFBE.6724.5AC0.8ED0" serial="DA0D.F464.0BCE.35B8.C0C0.28B6.D921.F3FD" key_number="VZ.02634184.0070"
+* Activation Key="A00E00-A0WC02-W1A863-482N41-BQAY84"
+=== Notes ===
+* our latest virt
+* Temp server to offload Virt13 till we can get a Cloud going.
+* virt 13 and 14 currently the only 64bit vz hosts
+== virt15 ==
+* Location: SHUTDOWN
+* OS: RedHat 9 x86
+* Networking: Priv IP: 10.1.4.65, Pub IP: 69.55.232.160 (2 onboard nics)
+* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: one 74 GB (2 x 74GB) RAID1 array, two 146 GB (2 x 146GB) RAID1 arrays, running on an LSI MegaRAID SCSI 320-1 RAID card.
+* Remote management: none
+* Virtuozzo version: 2.6.2
+* VZ license: hwid=A90F.6F48.E723.D8BA.3025.184A.5B73.D11E serial=E94B.5164.C1E6.A67F.67D1.7D96.0B6C.5524
+== virt16 ==
+* Location: castle, cab 3-7
+* OS: Fedora Core 4 x86
+* Networking: Priv IP: 10.1.4.66, Pub IP: 69.55.232.2 (2 onboard nics)
+* Hardware: Supermicro (custom build). 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: one 74 GB (2 x 74GB) RAID1 array, one 146 GB (2 x 146GB) RAID1 array, running on an LSI MegaRAID SCSI 320-1 RAID card.
+* Remote management: none
+* Virtuozzo version: 3.0.0
+* VZ license: hwid=DEFA.A325.7230.BBC8.9715.8B52.3FD7.27BE serial=66C0.41EA.3FBB.11D3.9CC6.55C7.09AE.14AB
+== virt17 ==
+* Location: castle, cab 3-6
+* OS: CentOS 4.4 x86
+* Networking: Priv IP: 10.1.4.67, Pub IP: 69.55.232.162, 2 onboard nics
+* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: two 146 GB (2 x 146GB) RAID1 arrays running on an LSI-based, Dell-branded (perc 5/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.217
+* Virtuozzo version: 3.0.0
+* VZ license: hwid=2E14.AED9.70B8.C26E.D99F.B0D3.BCD2.229C serial=2A11.DAD0.61DB.E889.8DF4.9AF7.CF82.3C37
+== virt19 ==
+* Location: castle, cab 3-6
+* OS: CentOS 5.2 x86
+* Networking: Priv IP: 10.1.4.69, Pub IP: 69.55.236.2, 2 onboard nics
+* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: one 146 GB (2 x 146GB) RAID1 array, one 300 GB (2 x 300GB) RAID1 array, running on an LSI-based, Dell-branded (perc 5/i) RAID card.
+* Remote management: [[DRAC/RMM|DRAC]] @ 10.1.4.219
+* Virtuozzo version: 3.0.0
+* VZ license: hwid=3968.13F7.B2AC.8952.8E19.13A9.6EF5.5822 serial=061D.84CD.CCE5.B213.15B5.C061.D6A7.B034
 = mail =
@@ Line 67: / Line 320: @@
 This machine (mail) is the swiss army knife of the company, playing host to many services and functions.
-* Location: castle, cab 3-8
+* Location: castle, cab 3-7
 * OS: FreeBSD 4.10 x86
 * Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
@@ Line 79: / Line 332: @@
 * bigbrother server/pager
 * snmp
+* named in a jail (ns1c)
 == email ==
@@ Line 144: / Line 398: @@
 <tt>tech1on@johncompanies.com
 tech1off@johncompanies.com</tt>
+== IP Blocking ==
+<pre>
+deny ip from 188.92.72.5 to any
+deny ip from any to 122.49.31.50
+deny ip from 122.49.31.50 to any
+deny ip from 74.208.225.225 to any
+deny ip from any to 216.243.118.35
+deny ip from 216.243.118.35 to any
+deny ip from any to 216.243.118.36
+deny ip from 216.243.118.36 to any
+deny ip from 112.215.0.0/18 to any   2014-08-13 Blocked PT Excelcomindo Pratama (Indonesia) for fradulent credit card attempts
+deny ip from 112.215.64.0/20 to any  2014-08-13 Blocked PT Excelcomindo Pratama (Indonesia) for fradulent credit card attempts
+deny ip from 120.168.0.0/24 to any   2014-08-13 Blocked Indosat 3G Broadband (Indonesia) for fradulent credit card attempts
+deny ip from 120.175.213.0/24 to any 2014-08-13 Blocked Indosat 3G Broadband (Indonesia) for fradulent credit card attempts
+</pre>
 == web ==
@@ Line 259: / Line 531: @@
 *[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
-= nat =
+== Building a new Mail Server ==
-== Summary ==
-This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.
+=== Installations ===
+I used FreeBSD 11.2
+The order is important especially for the Web Server.
+==== Web Server ====
-* Location: castle, cab 3-7
+I used FreeBSD 11.2
-* OS: FreeBSD 4.10 x86
+  perl 5.26
-* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
+  OpenSSL 1.0.2o-freebsd
-* Hardware: Custom 1U. single power supply.
+  pcre
-* Drives: one 8 GB IDE drive
+  apache22
+  mod_perl2
+  PayflowPro
+  mariadb 55 server and client
-== Services Provided ==
+Installation order is important
-* nat
-== nat control ==
+install perl 5.26.2 from ports
-All rules are contained in and look like:
-<pre>cat /etc/ipnat.rules
+<ore>
-# www (was 69.55.230.12)
+cd /usr/ports/lang/perl5.26/
-# virt19
+make
-#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
+[X] PERL_64BITINT  Use 64 bit integers (on i386)
-# virt18
+[X] USE_PERL       Rewrite links in /usr/bin
-#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
+(the rest unchecked
-# virt13
+make install
-#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
+</pre>
-# virt12
-#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
-# virt17
-bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
-# virt11
-#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
-# ASA
-#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
-# P1A
-bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
-#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
-# developer (was 69.55.230.17)
-# jail2
-#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
-# jail8
-#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
-# jail9
-#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
-# POLL
-#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
-# 1U SUN
-#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
-# ??
-#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
-# random machine
-#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
-#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
-# OFFICE OUTBOUND TRAFFIC
-#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
-#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>
-A simple entry looks like:
+install OpenSSL 1.0.2o-freebsd
- bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
-Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197
-To reload new rule config:
+<pre>
- ipnat -C -F -f /etc/ipnat.rules
+cd /usr/ports/
+make install
+</pre>
-You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.
+install pcre
-= nat2 =
+<pre>
-== Summary ==
+cd /usr/ports/
-This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.
+make install
+</pre>
-* Location: i2b, cab 6
-* OS: FreeBSD 6.4 x86
-* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
-* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
-* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.
-== Services Provided ==
+install Apache22
-* nat
-* bigbrother
-* ns3c (jail)
-* ntp
-== nat config ==
+<pre>
-Here's what's currently nat'd on nat2:
+cd /usr/ports/distfiles
-<pre>cat /etc/ipnat.rules
+fetch http://archive.apache.org/dist/httpd/httpd-2.2.32.tar.gz
-# sample entry
-#ATS-9
-bimap em0 10.1.2.79/32 -> 66.181.18.14/32
-#ATS-8
-bimap em0 10.1.2.78/32 -> 66.181.18.13/32
-#ATS-7
-bimap em0 10.1.2.77/32 -> 66.181.18.12/32
-#ATS-6
-bimap em0 10.1.2.76/32 -> 66.181.18.6/32
-#ATS-5
-bimap em0 10.1.2.75/32 -> 66.181.18.7/32
-#ATS-4
-bimap em0 10.1.2.74/32 -> 66.181.18.8/32
-#ATS-3
-bimap em0 10.1.2.73/32 -> 66.181.18.9/32
-#ATS-2
-bimap em0 10.1.2.72/32 -> 66.181.18.10/32
-#ATS-1
-bimap em0 10.1.2.71/32 -> 66.181.18.11/32
-#bwdb2
-bimap em0 10.1.2.4/32 -> 66.181.18.5/32
-# spare
-map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
+cd /usr/ports/www/apache22/tmp
+fetch --no-verify-peer http://mirror.nexcess.net/apache//httpd/httpd-2.2.34.tar.gz
+tar xvzf httpd-2.2.34.tar.gz
+./configure --prefix=/usr/local/apache --with-ssl=/usr/local/openssl/ --enable-ssl --enable-so --with-mpm=prefork --enable-threads --enable-mods-shared='mime alias setenvif dir' --enable-modules='mime alias setenvif dir' --with-pcre=/usr/local
+make install
+apachectl restart
-#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
+cd /usr/ports/www/apache22
-#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
+echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf
+make DISABLE_VULNERABILITIES=yes
+make install
 </pre>
-== build ==
-* partition map:
+install mod_perl2
-<pre>/ 512m
-swap 1G
-/var 256m
-/tmp 256m
-/usr 5g
-/mnt/data1 ~</pre>
-* edit /etc/make.conf
+<pre>
-<pre>echo "WITHOUT_X11=yes \
+cd /usr/ports/www/mod_perl2
-KERNCONF=nat2 \
+echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf
-BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>
+make DISABLE_VULNERABILITIES=yes
+make install
+</pre>
-* add settings to /boot/loader.conf and /boot.config
-<pre>echo "-Dh" >> /boot.config
+install mariadb
-echo 'console="comconsole,vidconsole" \
+<pre>
-boot_multicons="YES" \
+cd /usr/ports/databases/mariadb-103-server
-boot_serial="YES" \
+echo "DEFAULT_VERSIONS+=apache=2.2" >> /etc/make.conf
-comconsole_speed="115200"' >> /boot/loader.conf</pre>
+make DISABLE_VULNERABILITIES=yes
+make install
-* turn off all ttyv's except 0 and 1 in /etc/ttys
-also turn on ttyd0, change type to vt100:
-<pre>vi /etc/ttys
-ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
+</pre>
-ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
-ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
-ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
-ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
-ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
-# Serial terminals
-# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
-ttyd0   "/usr/libexec/getty std.9600"   vt100   on  secure
-kill -1 1</pre>
+==== Mail Server ====
-on console server:
+I used Postfix for email
- vi /etc/remote
-(rename port to jail8 depending on where and which digi plugged into)
-test serial console
+==== DNS Server (ns1c.johncompanies.com) ====
-* populate hosts
+= ns2c =
-<pre>echo "69.55.230.10 backup2" >> /etc/hosts
+== Summary ==
-echo "69.55.230.11 backup1" >> /etc/hosts
-echo "10.1.2.4 bwdb2" >> /etc/hosts
-echo "10.1.2.3 backup3" >> /etc/hosts</pre>
-* put key in authorized_keys on backup3
+* Location: castle, on lamphost ganeti cloud
-<pre>cd
+* OS: FreeBSD 11.2 x86_64
-ssh-keygen -t dsa -b 1024</pre>
+* Networking: Pub IP: 69.55.230.3   Private access: gnt-instance console ns2c.johncompanies.com
-(default location, leave password blank)
+* Hardware: on ganeti cloud gn6.jcihosting.net  secondary gn1.jcihosting.net
+* CPU:  1
+* RAM: 1 GB
+* Drives: 10 GB
+* Remote management:  gnt-instance console ns2c.johncompanies.com
-<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
+= nat =
-cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
+== Summary ==
-cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>
+This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.
-confirm that you can ssh to backup3 and backup 2 without getting a login prompt
+* Location: castle, cab 3-7
+* OS: FreeBSD 9.1 i386
+* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
+* Hardware: Custom 1U. single power supply.
+* Drives: one 8 GB IDE drive
- ssh backup3 hostname
+== Services Provided ==
+* nat
- ssh backup2 hostname
+== nat control ==
+All rules are contained in and look like:
- ssh backup1 hostname
+<pre>cat /etc/ipnat.rules
+# www (was 69.55.230.12)
+# virt19
-* edit root's path and login script:
+#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
- vi /root/.cshrc
+# virt18
+#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
-Change alias entries (add G):
+# virt13
-<pre>alias la        ls -aG
+#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
-alias lf        ls -FAG
+# virt12
-alias ll        ls -lAG
+#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
-alias ls        ls -AG
+# virt17
-alias mbm       mb mount
+bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
-alias mbu       mb umount</pre>
+# virt11
+#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
+# ASA
+#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
+# P1A
+bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
+#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
+# developer (was 69.55.230.17)
+# jail2
+#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
+# jail8
+#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
+# jail9
+#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
+# POLL
+#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
+# 1U SUN
+#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
+# ??
+#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
+# random machine
+#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
+#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
+# OFFICE OUTBOUND TRAFFIC
+#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
+#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>
+A simple entry looks like:
+ bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
+Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197
-and alter the prompt, set the following:
+To reload new rule config:
-  set prompt = "`/bin/hostname -s` %/# "
+  ipnat -C -F -f /etc/ipnat.rules
-* install cvsup
+You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.
-<pre>cd /usr/ports/net/cvsup-without-gui
-make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>
-* get latest sources for this release:
+= nat2 =
-<pre>cd /usr/src
+== Summary ==
-echo "*default host=cvsup4.freebsd.org\
+This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.
-*default base=/usr\
-*default prefix=/usr\
-*default release=cvs tag=RELENG_6_4\
-*default delete use-rel-suffix\
-*default compress\
-src-all" > sup
-cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>
+* Location: i2b, cab 6
+* OS: FreeBSD 6.4 x86
+* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
+* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
+* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.
+== Services Provided ==
+* nat
+* bigbrother
+* ns3c (jail)
+* wiki (jail)
+* ntp
-* configure new kernel.
+== nat config ==
+Here's what's currently nat'd on nat2:
- cd /usr/src/sys/i386/conf
+<pre>cat /etc/ipnat.rules
- scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2
+# sample entry
+#ATS-9
-* build, install kernel and world
+bimap em0 10.1.2.79/32 -> 66.181.18.14/32
+#ATS-8
-<pre>cd /boot
+bimap em0 10.1.2.78/32 -> 66.181.18.13/32
+#ATS-7
-mv kernel kernel.GENERIC
+bimap em0 10.1.2.77/32 -> 66.181.18.12/32
-cd kernel.GENERIC
+#ATS-6
-cd /usr/src
+bimap em0 10.1.2.76/32 -> 66.181.18.6/32
-make buildkernel installkernel
+#ATS-5
+bimap em0 10.1.2.75/32 -> 66.181.18.7/32
+#ATS-4
+bimap em0 10.1.2.74/32 -> 66.181.18.8/32
+#ATS-3
+bimap em0 10.1.2.73/32 -> 66.181.18.9/32
+#ATS-2
+bimap em0 10.1.2.72/32 -> 66.181.18.10/32
+#ATS-1
+bimap em0 10.1.2.71/32 -> 66.181.18.11/32
+#bwdb2
+bimap em0 10.1.2.4/32 -> 66.181.18.5/32
+# spare
+map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
-make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
+#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
-make installworld
+#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
-mergemaster -i
 </pre>
-* populate /etc/rc.conf with IPs and NFS settings
+== build ==
-<pre>vi /etc/rc.conf
-hostname="nat2.johncompanies.com"
+* partition map:
-kern_securelevel_enable="NO"
+<pre>/ 512m
-portmap_enable="NO"
+swap 1G
-sendmail_enable="NO"
+/var 256m
-usbd_enable="YES"
+/tmp 256m
+/usr 5g
+/mnt/data1 ~</pre>
-xntpd_enable="YES"
+* edit /etc/make.conf
-xntpd_flags="-A -p /var/run/ntpd.pid"
+<pre>echo "WITHOUT_X11=yes \
+KERNCONF=nat2 \
+BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>
-nfs_client_enable="YES"
+* add settings to /boot/loader.conf and /boot.config
-nfs_reserved_port_only="YES"
-ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
-#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
-#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
-ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
-ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
-ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
-defaultrouter="10.1.6.1"
-#defaultrouter=" 66.181.14.250"
-snmpd_enable="YES"
-ipnat_enable="YES"
-ipnat_rules="/etc/ipnat.rules"
-gateway_enable="YES"
-inetd_enable="YES"
+<pre>echo "-Dh" >> /boot.config
-inetd_flags="-wW -a 10.1.2.1"
-fsck_y_enable="YES"
-background_fsck="NO"
-sshd_enable="YES"</pre>
-* reboot. Confirm new kernel is loaded
+echo 'console="comconsole,vidconsole" \
+boot_multicons="YES" \
+boot_serial="YES" \
+comconsole_speed="115200"' >> /boot/loader.conf</pre>
- uname -a
+* turn off all ttyv's except 0 and 1 in /etc/ttys
+also turn on ttyd0, change type to vt100:
+<pre>vi /etc/ttys
-* update ports:
+ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
-<pre>cd /usr/ports
+ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
-echo "*default host=cvsup4.FreeBSD.org\
+ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
-*default base=/usr\
+ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
-*default prefix=/usr\
+ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
-*default release=cvs tag=RELENG_6_4\
+ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
-*default delete use-rel-suffix\
+# Serial terminals
-*default compress\
+# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
-ports-all tag=." > sup
+ttyd0   "/usr/libexec/getty std.9600"   vt100   on  secure
+kill -1 1</pre>
+on console server:
+ vi /etc/remote
+(rename port to jail8 depending on where and which digi plugged into)
+test serial console
-cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>
-* Install raid mgmt tool
+* populate hosts
+<pre>echo "69.55.230.10 backup2" >> /etc/hosts
+echo "69.55.230.11 backup1" >> /etc/hosts
+echo "10.1.2.4 bwdb2" >> /etc/hosts
+echo "10.1.2.3 backup3" >> /etc/hosts</pre>
-<pre>cd /usr/local/sbin
+* put key in authorized_keys on backup3
-fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
+<pre>cd
-tar xzf tw_cli-freebsd-x86-9.5.3.tgz
+ssh-keygen -t dsa -b 1024</pre>
-rm tw_cli-freebsd-x86-9.5.3.tgz
+(default location, leave password blank)
-chmod 0700 tw_cli</pre>
-Test:
+<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
- ./tw_cli info c0
+cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
+cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>
-* install rsync from ports
+confirm that you can ssh to backup3 and backup 2 without getting a login prompt
-<pre>cd /usr/ports/net/rsync
-make install clean</pre>
-choose default options
-* install perl from ports
+ ssh backup3 hostname
-<pre>cd /usr/ports/lang/perl5.8
-make install clean</pre>
-* install screen from ports
+ ssh backup2 hostname
-<pre>cd /usr/ports/sysutils/screen
-make install clean</pre>
-* install bb client
+  ssh backup1 hostname
-<pre>adduser
-Username: bb
-Full name: bb
-Uid (Leave empty for default): 1984
-Login group [bb]:
-Login group is bb. Invite bb into other groups? []:
-Login class [default]:
-Shell (sh csh tcsh nologin) [sh]:
-Home directory [/home/bb]:
-Use password-based authentication? [yes]:
-Use an empty password? (yes/no) [no]:
-Use a random password? (yes/no) [no]: yes
-Lock out the account after creation? [no]:
-Username   : bb
-Password   : <random>
-Full Name  : bb
-Uid        : 1984
-Class      :
-Groups     : bb
-Home       : /home/bb
-Shell      : /bin/sh
-Locked     : no
-OK? (yes/no): yes
-cd /usr/home/bb
-scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
-tar xvf bb-freebsd.tar</pre>
-edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
-<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
-.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
-vi /home/bb/bbc1.9e-btf/ext/openfiles
-MACHINE="nat2,johncompanies,com"      # HAS TO BE IN A,B,C FORM
-cd /usr/home/bb/bbc1.9e-btf/etc
+* edit root's path and login script:
-./bbchkcfg.sh
+ vi /root/.cshrc
-(y to questions)
-./bbchkhosts.sh
-(ignore ssh errors)
-cd ../..
-chown -R bb .
-su bb
-cd
-cd bbc1.9e-btf/src
-make; make install
-cd ..
-vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
+Change alias entries (add G):
-        $1 $TOPARGS > $BBTMP/TOP.$$
+<pre>alias la        ls -aG
-#        /usr/local/jail/bin/jtop > $BBTMP/TOP.$$
+alias lf        ls -FAG
+alias ll        ls -lAG
+alias ls        ls -AG
+alias mbm       mb mount
+alias mbu       mb umount</pre>
+and alter the prompt, set the following:
+ set prompt = "`/bin/hostname -s` %/# "
-./runbb.sh start
+* install cvsup
-more BBOUT
+<pre>cd /usr/ports/net/cvsup-without-gui
-(look for errors)
+make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>
-exit
-echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
+* get latest sources for this release:
-chmod +x /usr/local/etc/rc.d/bb.sh
+<pre>cd /usr/src
-</pre>
+echo "*default host=cvsup4.freebsd.org\
+*default base=/usr\
+*default prefix=/usr\
+*default release=cvs tag=RELENG_6_4\
+*default delete use-rel-suffix\
+*default compress\
+src-all" > sup
-Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):
+cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>
- ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2
-* configure bb on mail:
+* configure new kernel.
-<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
-.163.14.48 nat2.johncompanies.com # ssh
-su bb
+ cd /usr/src/sys/i386/conf
-cd
+ scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2
-bbsrc/bb/runbb.sh restart ; exit</pre>
-* configure ntp
+* build, install kernel and world
-<pre>echo "server 69.55.230.2
-server 0.pool.ntp.org
-server 1.pool.ntp.org
-server 2.pool.ntp.org
-server 3.pool.ntp.org
-server 66.187.233.4
-server 217.204.76.170
-server 64.112.189.11
-server 66.69.112.130
-server 80.85.129.25
-server 80.237.234.15
-server 130.60.7.44
-server 134.99.176.3
-server 198.144.202.250
-server 202.74.170.194
-server 204.17.42.199
-server 204.87.183.6
-server 213.15.3.1
-server 213.239.178.33
-server 217.114.97.97
-server 69.55.230.2" > /etc/ntp.conf</pre>
-<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
+<pre>cd /boot
-sleep 2; ntpq -p</pre>
-(confirm it’s able to reach our time server)
- echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
+mv kernel kernel.GENERIC
- chmod 0700 /usr/local/etc/rc.d/ntp.sh
+cd kernel.GENERIC
+cd /usr/src
+make buildkernel installkernel
-* fwd and reverse lookups on ns1c
+make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
- vr johncompanies.com
+make installworld
- (edit the PTR too)
+mergemaster -i
+</pre>
-* setup backups, nfs mount
+* populate /etc/rc.conf with IPs and NFS settings
+<pre>vi /etc/rc.conf
-<pre>mkdir /backup3
+hostname="nat2.johncompanies.com"
-echo 'backup3:/data           /backup3        nfs     rw,bg           0       0' >> /etc/fstab
+kern_securelevel_enable="NO"
+portmap_enable="NO"
+sendmail_enable="NO"
+usbd_enable="YES"
-echo '#\!/bin/sh\
+xntpd_enable="YES"
-backupdir=/data/nat2/current\
+xntpd_flags="-A -p /var/run/ntpd.pid"
-\
-## ENTRY /etc ' > /usr/local/etc/backup.config</pre>
-on backup3:
+nfs_client_enable="YES"
-setup backup dirs:
+nfs_reserved_port_only="YES"
- ssh backup3 mkdir -p /data/nat2/current
+ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
+#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
+#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
+ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
+ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
+ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
+defaultrouter="10.1.6.1"
+#defaultrouter=" 66.181.14.250"
+snmpd_enable="YES"
+ipnat_enable="YES"
+ipnat_rules="/etc/ipnat.rules"
+gateway_enable="YES"
-on backup3, add the system to
+inetd_enable="YES"
- vi /usr/local/sbin/snapshot_archive
+inetd_flags="-wW -a 10.1.2.1"
+fsck_y_enable="YES"
+background_fsck="NO"
+sshd_enable="YES"</pre>
-<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
+* reboot. Confirm new kernel is loaded
-vi /usr/local/etc/rsync.backup
-backup1 > backup3</pre>
-  crontab -e
+  uname -a
-0 * * * /usr/local/etc/rsync.backup
-* edit sshd_config for security
+* update ports:
-<pre>vi /etc/ssh/sshd_config
+<pre>cd /usr/ports
-ListenAddress 66.181.18.1
+echo "*default host=cvsup4.FreeBSD.org\
-ListenAddress 69.55.229.2
+*default base=/usr\
-ListenAddress 10.1.2.1
+*default prefix=/usr\
+*default release=cvs tag=RELENG_6_4\
+*default delete use-rel-suffix\
+*default compress\
+ports-all tag=." > sup
-kill -1 `cat /var/run/sshd.pid`</pre>
+cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>
-* raid chk
+* Install raid mgmt tool
-<pre>cat > /usr/local/sbin/lsiraidchk
+<pre>cd /usr/local/sbin
-#!/usr/bin/perl
+fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
+tar xzf tw_cli-freebsd-x86-9.5.3.tgz
+rm tw_cli-freebsd-x86-9.5.3.tgz
+chmod 0700 tw_cli</pre>
-my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;
+Test:
+ ./tw_cli info c0
-foreach (@out) {
+* install rsync from ports
-    if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
+<pre>cd /usr/ports/net/rsync
-#print $_;
+make install clean</pre>
-}</pre>
+choose default options
-* netflow stuff
+* install perl from ports
-add crontab entries
+<pre>cd /usr/ports/lang/perl5.8
-<pre>crontab -e
+make install clean</pre>
-3 * * * /usr/local/etc/rsync.backup
-0 1 * * /sbin/ipfw zero
-0 1 * * /sbin/ipfw del 3 4 5
-23 30 * * /sbin/ipfw show > /tmp/ipfw_count
-0 30 * * /sbin/ipfw show > /tmp/ipfw_count
-3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
-*/5 * * * * /usr/local/sbin/lsiraidchk
-#10 0 * * * rm /var/spool/clientmqueue/*</pre>
+* install screen from ports
+<pre>cd /usr/ports/sysutils/screen
+make install clean</pre>
-<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
+* install bb client
-scp /etc/makepiperules.pl user@64.163.14.48:~
+<pre>adduser
-mv /home/user/makefwrules.pl /etc
+Username: bb
-mv /home/user/makepiperules.pl /etc
+Full name: bb
-touch /etc/firewall.sh
+Uid (Leave empty for default): 1984
-mkdir /etc/oldrules/</pre>
+Login group [bb]:
+Login group is bb. Invite bb into other groups? []:
-other binaries
+Login class [default]:
+Shell (sh csh tcsh nologin) [sh]:
-<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
+Home directory [/home/bb]:
-mv ~user/rulemaker /usr/local/sbin
+Use password-based authentication? [yes]:
-scp ~user/Sendmail.pm user@64.163.14.48:~
+Use an empty password? (yes/no) [no]:
-scp ~user/doswatch.pl user@64.163.14.48:~</pre>
+Use a random password? (yes/no) [no]: yes
+Lock out the account after creation? [no]:
+Username   : bb
+Password   : <random>
+Full Name  : bb
+Uid        : 1984
+Class      :
+Groups     : bb
+Home       : /home/bb
+Shell      : /bin/sh
+Locked     : no
+OK? (yes/no): yes
-* add nat rules
+cd /usr/home/bb
-<pre>vi /etc/ipnat.rules
+scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
-# sample entry
+tar xvf bb-freebsd.tar</pre>
-bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
+edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
-#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
+<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
+.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
+vi /home/bb/bbc1.9e-btf/ext/openfiles
+MACHINE="nat2,johncompanies,com"      # HAS TO BE IN A,B,C FORM
-ipnat -C -f /etc/ipnat.rules</pre>
+cd /usr/home/bb/bbc1.9e-btf/etc
+./bbchkcfg.sh
+(y to questions)
+./bbchkhosts.sh
+(ignore ssh errors)
+cd ../..
+chown -R bb .
+su bb
+cd
+cd bbc1.9e-btf/src
+make; make install
+cd ..
-* shell for user
+vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
-<pre>cp /root/.cshrc ~user/
+        $1 $TOPARGS > $BBTMP/TOP.$$
-vi ~user/</pre>
+#        /usr/local/jail/bin/jtop > $BBTMP/TOP.$$
-change # to $
-* mrtg
+./runbb.sh start
+more BBOUT
+(look for errors)
+exit
-<pre>cd /usr/ports/net-mgmt/mrtg
+echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
-make install clean</pre>
+chmod +x /usr/local/etc/rc.d/bb.sh
-(no FONTCONFIG, v3)
+</pre>
-this didn't work cause of libtool incompat
+Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):
-so manually moved files:
+ ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2
-<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
+* configure bb on mail:
-scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
+<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
-scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
+.163.14.48 nat2.johncompanies.com # ssh
-scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
-scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
-scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
-scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
-scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
-scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
-rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
-scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
-scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
-rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
-scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4
-rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
+su bb
-cd /usr/ports/net/rrdtool
+cd
-make install
+bbsrc/bb/runbb.sh restart ; exit</pre>
-mkdir -p /mnt/data1/mrtg/data
+* configure ntp
-scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
+<pre>echo "server 69.55.230.2
-scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/
+server 0.pool.ntp.org
+server 1.pool.ntp.org
-cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg
+server 2.pool.ntp.org
+server 3.pool.ntp.org
-cat > /mnt/data1/mrtg/mrtg.sh
+server 66.187.233.4
-#!/bin/sh
+server 217.204.76.170
-/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg
+server 64.112.189.11
+server 66.69.112.130
-chmod 0700 /mnt/data1/mrtg/mrtg.sh
+server 80.85.129.25
+server 80.237.234.15
-crontab -e
+server 130.60.7.44
-* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>
+server 134.99.176.3
+server 198.144.202.250
+server 202.74.170.194
+server 204.17.42.199
+server 204.87.183.6
+server 213.15.3.1
+server 213.239.178.33
+server 217.114.97.97
+server 69.55.230.2" > /etc/ntp.conf</pre>
-* snmp firewall block
+<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
-<pre>cat > /usr/local/etc/rc.d/boot.sh
+sleep 2; ntpq -p</pre>
-ipfw add 10 allow udp from 69.55.230.2 to any 161
+(confirm it’s able to reach our time server)
-ipfw add 10 allow udp from 10.1.2.1 to any 161
-ipfw add 11 deny udp from any to any 161
-chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>
-= bwdb =
+ echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
-== Summary ==
+ chmod 0700 /usr/local/etc/rc.d/ntp.sh
-This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.
-* Location: castle, cab 3-7
+* fwd and reverse lookups on ns1c
-* OS: FreeBSD 4.10 x86
+ vr johncompanies.com
-* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
+ (edit the PTR too)
-* Hardware: Custom 1U. Single power supply.
-* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.
-== Services Provided ==
+* setup backups, nfs mount
-* netflow
-* mysql
-* bigbrother
-* snmp
-== netflow ==
+<pre>mkdir /backup3
+echo 'backup3:/data           /backup3        nfs     rw,bg           0       0' >> /etc/fstab
-The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.
+echo '#\!/bin/sh\
+backupdir=/data/nat2/current\
+\
+## ENTRY /etc ' > /usr/local/etc/backup.config</pre>
+on backup3:
+setup backup dirs:
+ ssh backup3 mkdir -p /data/nat2/current
+on backup3, add the system to
+ vi /usr/local/sbin/snapshot_archive
-A cronjob moves that flow file (or files if there are multiple due to some delay)
+<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
-,16,31,46 * * * * /usr/home/flowbin/queue.pl
+vi /usr/local/etc/rsync.backup
+backup1 > backup3</pre>
+ crontab -e
+0 * * * /usr/local/etc/rsync.backup
+* edit sshd_config for security
+<pre>vi /etc/ssh/sshd_config
+ListenAddress 66.181.18.1
+ListenAddress 69.55.229.2
+ListenAddress 10.1.2.1
-into a processing queue:
+kill -1 `cat /var/run/sshd.pid`</pre>
-<tt>/usr/home/working</tt>
-Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
+* raid chk
-,17,32,47 * * * * /usr/home/flowbin/processflows.pl
-== mysql ==
+<pre>cat > /usr/local/sbin/lsiraidchk
+#!/usr/bin/perl
-The database storing all the traffic data is named <tt>traffic</tt>
+my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;
-Tables:
-<pre>mysql> show tables;
+foreach (@out) {
-+---------------------------+
+    if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
-| Tables_in_traffic         |
+#print $_;
-+---------------------------+
+}</pre>
-| dailyIpTotals_69_55_224   |
-| dailyIpTotals_69_55_225   |
+* netflow stuff
-| dailyIpTotals_69_55_226   |
+add crontab entries
-| dailyIpTotals_69_55_227   |
+<pre>crontab -e
-| dailyIpTotals_69_55_228   |
+3 * * * /usr/local/etc/rsync.backup
-| dailyIpTotals_69_55_229   |
+0 1 * * /sbin/ipfw zero
-| dailyIpTotals_69_55_230   |
+0 1 * * /sbin/ipfw del 3 4 5
-| dailyIpTotals_69_55_231   |
+23 30 * * /sbin/ipfw show > /tmp/ipfw_count
-| dailyIpTotals_69_55_232   |
+0 30 * * /sbin/ipfw show > /tmp/ipfw_count
-| dailyIpTotals_69_55_233   |
+3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
-| dailyIpTotals_69_55_234   |
+*/5 * * * * /usr/local/sbin/lsiraidchk
-| dailyIpTotals_69_55_235   |
-| dailyIpTotals_69_55_236   |
+#10 0 * * * rm /var/spool/clientmqueue/*</pre>
-| dailyIpTotals_69_55_237   |
-| dailyIpTotals_69_55_238   |
+<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
-| dailyIpTotals_69_55_239   |
+scp /etc/makepiperules.pl user@64.163.14.48:~
-| dailyPortTotals_69_55_224 |
+mv /home/user/makefwrules.pl /etc
-| dailyPortTotals_69_55_225 |
+mv /home/user/makepiperules.pl /etc
-| dailyPortTotals_69_55_226 |
+touch /etc/firewall.sh
-| dailyPortTotals_69_55_227 |
+mkdir /etc/oldrules/</pre>
-| dailyPortTotals_69_55_228 |
-| dailyPortTotals_69_55_229 |
-| dailyPortTotals_69_55_230 |
-| dailyPortTotals_69_55_231 |
-| dailyPortTotals_69_55_232 |
-| dailyPortTotals_69_55_233 |
-| dailyPortTotals_69_55_234 |
-| dailyPortTotals_69_55_235 |
-| dailyPortTotals_69_55_236 |
-| dailyPortTotals_69_55_237 |
-| dailyPortTotals_69_55_238 |
-| dailyPortTotals_69_55_239 |
-| ipTotals_69_55_224        |
-| ipTotals_69_55_225        |
-| ipTotals_69_55_226        |
-| ipTotals_69_55_227        |
-| ipTotals_69_55_228        |
-| ipTotals_69_55_229        |
-| ipTotals_69_55_230        |
-| ipTotals_69_55_231        |
-| ipTotals_69_55_232        |
-| ipTotals_69_55_233        |
-| ipTotals_69_55_234        |
-| ipTotals_69_55_235        |
-| ipTotals_69_55_236        |
-| ipTotals_69_55_237        |
-| ipTotals_69_55_238        |
-| ipTotals_69_55_239        |
-| portTotals_69_55_224      |
-| portTotals_69_55_225      |
-| portTotals_69_55_226      |
-| portTotals_69_55_227      |
-| portTotals_69_55_228      |
-| portTotals_69_55_229      |
-| portTotals_69_55_230      |
-| portTotals_69_55_231      |
-| portTotals_69_55_232      |
-| portTotals_69_55_233      |
-| portTotals_69_55_234      |
-| portTotals_69_55_235      |
-| portTotals_69_55_236      |
-| portTotals_69_55_237      |
-| portTotals_69_55_238      |
-| portTotals_69_55_239      |
-+---------------------------+
-</pre>
-So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.
+other binaries
-<pre>mysql> describe dailyIpTotals_69_55_224;
+<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
-+-----------+-------------+------+-----+---------+-------+
+mv ~user/rulemaker /usr/local/sbin
-| Field     | Type        | Null | Key | Default | Extra |
+scp ~user/Sendmail.pm user@64.163.14.48:~
-+-----------+-------------+------+-----+---------+-------+
+scp ~user/doswatch.pl user@64.163.14.48:~</pre>
-| id        | varchar(23) |      | PRI |         |       |
-| date      | date        | YES  |     | NULL    |       |
+* add nat rules
-| ip        | varchar(15) | YES  | MUL | NULL    |       |
+<pre>vi /etc/ipnat.rules
-| direction | tinyint(1)  | YES  |     | NULL    |       |
+# sample entry
-| octets    | bigint(12)  | YES  |     | NULL    |       |
+bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
-| packets   | int(11)     | YES  |     | NULL    |       |
+#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
-+-----------+-------------+------+-----+---------+-------+
-mysql> select * from dailyIpTotals_69_55_224 limit 1\G
+ipnat -C -f /etc/ipnat.rules</pre>
-*************************** 1. row ***************************
-       id: 6955224194-20100917-1
-     date: 2010-09-17
-       ip: 69.55.224.194
-direction: 1
-   octets: 8821
-  packets: 91
-</pre>
-The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.
+* shell for user
+<pre>cp /root/.cshrc ~user/
+vi ~user/</pre>
+change # to $
-The 15-minute table has similar information, but it's organized in 15 minute increments:
+* mrtg
-<pre>mysql> describe ipTotals_69_55_224;
+<pre>cd /usr/ports/net-mgmt/mrtg
-+-----------+------------+------+-----+---------+-------+
+make install clean</pre>
-| Field     | Type       | Null | Key | Default | Extra |
+(no FONTCONFIG, v3)
-+-----------+------------+------+-----+---------+-------+
-| date      | datetime   | YES  |     | NULL    |       |
+this didn't work cause of libtool incompat
-| ip        | char(15)   | YES  | MUL | NULL    |       |
-| direction | tinyint(1) | YES  |     | NULL    |       |
-| octets    | bigint(20) | YES  |     | NULL    |       |
-| packets   | int(11)    | YES  |     | NULL    |       |
-+-----------+------------+------+-----+---------+-------+
-mysql> select * from ipTotals_69_55_224 limit 2\G
+so manually moved files:
-*************************** 1. row ***************************
-     date: 2010-01-11 19:30:00
-       ip: 69.55.224.13
-direction: 1
-   octets: 288
-  packets: 6
-*************************** 2. row ***************************
-     date: 2010-01-11 19:30:00
-       ip: 69.55.224.12
-direction: 1
-   octets: 216
-  packets: 4</pre>
-So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.
+<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
+scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
+scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
+scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
+scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
+scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
+scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
+scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
+scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
+rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
+scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
+scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
+rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
+scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4
-That covers the simple traffic tabulation tables. We also track traffic by port:
+rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
+cd /usr/ports/net/rrdtool
+make install
-<pre>mysql> describe dailyPortTotals_69_55_224;
+mkdir -p /mnt/data1/mrtg/data
-+-----------+-------------+------+-----+---------+-------+
+scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
-| Field     | Type        | Null | Key | Default | Extra |
+scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/
-+-----------+-------------+------+-----+---------+-------+
-| id        | varchar(28) |      | PRI |         |       |
-| date      | date        | YES  |     | NULL    |       |
-| ip        | varchar(15) | YES  | MUL | NULL    |       |
-| direction | tinyint(1)  | YES  |     | NULL    |       |
-| protocol  | smallint(3) | YES  |     | NULL    |       |
-| port      | int(11)     | YES  |     | NULL    |       |
-| octets    | bigint(11)  | YES  |     | NULL    |       |
-| packets   | int(11)     | YES  |     | NULL    |       |
-+-----------+-------------+------+-----+---------+-------+
-rows in set (0.00 sec)
-mysql> select * from dailyPortTotals_69_55_224 limit 1\G
+cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg
-*************************** 1. row ***************************
-       id: 695522496-20091218-1-6-23
-     date: 2009-12-18
-       ip: 69.55.224.96
-direction: 1
- protocol: 6
-     port: 23
-   octets: 1796
-  packets: 30
-mysql> select * from portTotals_69_55_224 limit 1\G
+cat > /mnt/data1/mrtg/mrtg.sh
-*************************** 1. row ***************************
+#!/bin/sh
-     date: 2010-09-07 18:45:00
+/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg
-       ip: 69.55.224.254
-direction: 1
- protocol: 6
-     port: 99999
-   octets: 144
-  packets: 3
-</pre>
+chmod 0700 /mnt/data1/mrtg/mrtg.sh
-This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.
+crontab -e
+* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>
-== Regular maintenance ==
+* snmp firewall block
-*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
+<pre>cat > /usr/local/etc/rc.d/boot.sh
-* archive data from database
+ipfw add 10 allow udp from 69.55.230.2 to any 161
- archive_daily.pl 2012 09
+ipfw add 10 allow udp from 10.1.2.1 to any 161
-This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.
+ipfw add 11 deny udp from any to any 161
+chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>
+= bwdb =
+== Summary ==
+This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.
- archive_15min.pl 2012 09
+* Location: castle, cab 3-7
-This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.
+* OS: FreeBSD 4.10 x86
+* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
+* Hardware: Custom 1U. Single power supply.
+* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.
-* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively
+== Services Provided ==
+* netflow
+* mysql
+* bigbrother
+* snmp
-== Slaving ==
+== netflow ==
-If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:
+The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.
-On the traffic master:
+A cronjob moves that flow file (or files if there are multiple due to some delay)
+,16,31,46 * * * * /usr/home/flowbin/queue.pl
- GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';
+into a processing queue:
+<tt>/usr/home/working</tt>
-in my.cnf:
+Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
-<pre>bin-log
+,17,32,47 * * * * /usr/home/flowbin/processflows.pl
-server-id=1
-max_binlog_size=500M
-expire_logs_days = 3</pre>
-on slave:
+== mysql ==
-in my.cnf:
-<pre>server-id       = 2
-master-host     =   10.1.4.203
-master-user     =   repl
-master-password =   qERUG8wf
-master-connect-retry=60
-replicate-wild-do-table=traffic.daily%
-max_relay_log_size=500M
-expire_logs_days = 3
-replicate-wild-do-table=traffic.%</pre>
+The database storing all the traffic data is named <tt>traffic</tt>
+Tables:
-on master:
+<pre>mysql> show tables;
- touch /usr/home/working/.lock
++---------------------------+
-(make sure processflows not running)
+| Tables_in_traffic         |
++---------------------------+
-<pre>FLUSH TABLES WITH READ LOCK;
+| dailyIpTotals_69_55_224   |
-cd /usr/home/database/traffic
+| dailyIpTotals_69_55_225   |
-tar -czf mysql-traffic-snapshot.tgz ./daily*
+| dailyIpTotals_69_55_226   |
-(~1G)
+| dailyIpTotals_69_55_227   |
-SHOW MASTER STATUS;
+| dailyIpTotals_69_55_228   |
-+-----------------+-----------+--------------+------------------+
+| dailyIpTotals_69_55_229   |
-| File            | Position  | Binlog_Do_DB | Binlog_Ignore_DB |
+| dailyIpTotals_69_55_230   |
-+-----------------+-----------+--------------+------------------+
+| dailyIpTotals_69_55_231   |
-| bwdb-bin.000039 | 154432615 |              |                  |
+| dailyIpTotals_69_55_232   |
-+-----------------+-----------+--------------+------------------+
+| dailyIpTotals_69_55_233   |
+| dailyIpTotals_69_55_234   |
-(write down info)
+| dailyIpTotals_69_55_235   |
-UNLOCK TABLES;
+| dailyIpTotals_69_55_236   |
-scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>
+| dailyIpTotals_69_55_237   |
+| dailyIpTotals_69_55_238   |
-on slave:
+| dailyIpTotals_69_55_239   |
-<pre>mkdir /mnt/data1/db/mysql/traffic
+| dailyPortTotals_69_55_224 |
-cd /mnt/data1/db/mysql/traffic/
+| dailyPortTotals_69_55_225 |
-tar xzvf mysql-traffic-snapshot.tgz
+| dailyPortTotals_69_55_226 |
-(restart mysql)
+| dailyPortTotals_69_55_227 |
-CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
+| dailyPortTotals_69_55_228 |
-START SLAVE;</pre>
+| dailyPortTotals_69_55_229 |
+| dailyPortTotals_69_55_230 |
-<pre>cd /usr/home/database/traffic
+| dailyPortTotals_69_55_231 |
-scp *</pre>
+| dailyPortTotals_69_55_232 |
+| dailyPortTotals_69_55_233 |
-<pre>optimize table dailyPortTotals_69_55_224;
+| dailyPortTotals_69_55_234 |
-optimize table dailyPortTotals_69_55_225;
+| dailyPortTotals_69_55_235 |
-optimize table dailyPortTotals_69_55_226;
+| dailyPortTotals_69_55_236 |
-optimize table dailyPortTotals_69_55_227;
+| dailyPortTotals_69_55_237 |
-optimize table dailyPortTotals_69_55_228;
+| dailyPortTotals_69_55_238 |
-optimize table dailyPortTotals_69_55_229;
+| dailyPortTotals_69_55_239 |
-optimize table dailyPortTotals_69_55_230;
+| ipTotals_69_55_224        |
-optimize table dailyPortTotals_69_55_231;
+| ipTotals_69_55_225        |
-optimize table dailyPortTotals_69_55_232;
+| ipTotals_69_55_226        |
-optimize table dailyPortTotals_69_55_233;
+| ipTotals_69_55_227        |
-optimize table dailyPortTotals_69_55_234;
+| ipTotals_69_55_228        |
-optimize table dailyPortTotals_69_55_235;
+| ipTotals_69_55_229        |
-optimize table dailyPortTotals_69_55_236;
+| ipTotals_69_55_230        |
-optimize table dailyPortTotals_69_55_237;
+| ipTotals_69_55_231        |
-optimize table dailyPortTotals_69_55_238;
+| ipTotals_69_55_232        |
-optimize table dailyPortTotals_69_55_239;</pre>
+| ipTotals_69_55_233        |
+| ipTotals_69_55_234        |
+| ipTotals_69_55_235        |
+| ipTotals_69_55_236        |
+| ipTotals_69_55_237        |
+| ipTotals_69_55_238        |
+| ipTotals_69_55_239        |
+| portTotals_69_55_224      |
+| portTotals_69_55_225      |
+| portTotals_69_55_226      |
+| portTotals_69_55_227      |
+| portTotals_69_55_228      |
+| portTotals_69_55_229      |
+| portTotals_69_55_230      |
+| portTotals_69_55_231      |
+| portTotals_69_55_232      |
+| portTotals_69_55_233      |
+| portTotals_69_55_234      |
+| portTotals_69_55_235      |
+| portTotals_69_55_236      |
+| portTotals_69_55_237      |
+| portTotals_69_55_238      |
+| portTotals_69_55_239      |
++---------------------------+
+</pre>
-== Build ==
+So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.
-=== BIOS Config ===
+<pre>mysql> describe dailyIpTotals_69_55_224;
-disable quiet boot
++-----------+-------------+------+-----+---------+-------+
+| Field     | Type        | Null | Key | Default | Extra |
-set to last state after power loss
++-----------+-------------+------+-----+---------+-------+
+| id        | varchar(23) |      | PRI |         |       |
+| date      | date        | YES  |     | NULL    |       |
+| ip        | varchar(15) | YES  | MUL | NULL    |       |
+| direction | tinyint(1)  | YES  |     | NULL    |       |
+| octets    | bigint(12)  | YES  |     | NULL    |       |
+| packets   | int(11)     | YES  |     | NULL    |       |
++-----------+-------------+------+-----+---------+-------+
-set date/time to GMT
+mysql> select * from dailyIpTotals_69_55_224 limit 1\G
+*************************** 1. row ***************************
+       id: 6955224194-20100917-1
+     date: 2010-09-17
+       ip: 69.55.224.194
+direction: 1
+   octets: 8821
+  packets: 91
+</pre>
-enable serial console output (baud rate 115200)
+The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.
-=== Install OS ===
+The 15-minute table has similar information, but it's organized in 15 minute increments:
-Install FreeBSD 8.3 amd64
+<pre>mysql> describe ipTotals_69_55_224;
++-----------+------------+------+-----+---------+-------+
+| Field     | Type       | Null | Key | Default | Extra |
++-----------+------------+------+-----+---------+-------+
+| date      | datetime   | YES  |     | NULL    |       |
+| ip        | char(15)   | YES  | MUL | NULL    |       |
+| direction | tinyint(1) | YES  |     | NULL    |       |
+| octets    | bigint(20) | YES  |     | NULL    |       |
+| packets   | int(11)    | YES  |     | NULL    |       |
++-----------+------------+------+-----+---------+-------+
-* partition map:
+mysql> select * from ipTotals_69_55_224 limit 2\G
-<pre>/ 500m
+*************************** 1. row ***************************
-swap 4096m
+     date: 2010-01-11 19:30:00
-/var 256m
+       ip: 69.55.224.13
-/tmp 256m
+direction: 1
-/usr ~</pre>
+   octets: 288
+  packets: 6
+*************************** 2. row ***************************
+     date: 2010-01-11 19:30:00
+       ip: 69.55.224.12
+direction: 1
+   octets: 216
+  packets: 4</pre>
-* edit /etc/make.conf
+So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.
-Castle:
-<pre>echo "WITHOUT_X11=yes \
-KERNCONF=bwdb \
-BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>
-i2b:
+That covers the simple traffic tabulation tables. We also track traffic by port:
-<pre>echo "WITHOUT_X11=yes \
-KERNCONF=bwdb2 \
-BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>
-* add settings to /boot/loader.conf and /boot.config
+<pre>mysql> describe dailyPortTotals_69_55_224;
++-----------+-------------+------+-----+---------+-------+
+| Field     | Type        | Null | Key | Default | Extra |
++-----------+-------------+------+-----+---------+-------+
+| id        | varchar(28) |      | PRI |         |       |
+| date      | date        | YES  |     | NULL    |       |
+| ip        | varchar(15) | YES  | MUL | NULL    |       |
+| direction | tinyint(1)  | YES  |     | NULL    |       |
+| protocol  | smallint(3) | YES  |     | NULL    |       |
+| port      | int(11)     | YES  |     | NULL    |       |
+| octets    | bigint(11)  | YES  |     | NULL    |       |
+| packets   | int(11)     | YES  |     | NULL    |       |
++-----------+-------------+------+-----+---------+-------+
+rows in set (0.00 sec)
-<pre>echo "-Dh" >> /boot.config
+mysql> select * from dailyPortTotals_69_55_224 limit 1\G
+*************************** 1. row ***************************
+       id: 695522496-20091218-1-6-23
+     date: 2009-12-18
+       ip: 69.55.224.96
+direction: 1
+ protocol: 6
+     port: 23
+   octets: 1796
+  packets: 30
-echo 'console="comconsole,vidconsole" \
+mysql> select * from portTotals_69_55_224 limit 1\G
-boot_multicons="YES" \
+*************************** 1. row ***************************
-boot_serial="YES" \
+     date: 2010-09-07 18:45:00
-comconsole_speed="115200"' >> /boot/loader.conf</pre>
+       ip: 69.55.224.254
+direction: 1
+ protocol: 6
+     port: 99999
+   octets: 144
+  packets: 3
+</pre>
-* turn off all ttyv's except 0 and 1 in /etc/ttys
+This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.
-also turn on ttyu0, change type to vt100:
-<pre>vi /etc/ttys
-ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
+== Regular maintenance ==
-ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
+*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
-ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
+* archive data from database
-ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
+  archive_daily.pl 2012 09
-ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
+This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.
-ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
-# Serial terminals
-# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
-ttyu0   "/usr/libexec/getty std.9600"   vt100   on secure
-kill -1 1</pre>
+ archive_15min.pl 2012 09
+This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.
-on console server:
+* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively
- vi /etc/remote
-(rename port to jail8 depending on where and which digi plugged into)
-test serial console
+== Slaving ==
-* populate hosts
+If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:
-i2b:
-<pre>echo "69.55.230.10 backup2" >> /etc/hosts
-echo "69.55.230.11 backup1" >> /etc/hosts
-echo "10.1.2.3 backup3" >> /etc/hosts</pre>
-castle:
+On the traffic master:
-<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
-echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
-echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
-</pre>
-* put key in authorized_keys on backup1 and  backup2
+ GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';
- cd
- ssh-keygen -t dsa -b 1024
-(default location, leave password blank)
-castle:
+in my.cnf:
- cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
+<pre>bin-log
- cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
+server-id=1
+max_binlog_size=500M
+expire_logs_days = 3</pre>
-i2b:
+on slave:
- cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
+in my.cnf:
- cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
+<pre>server-id       = 2
- cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
+master-host     =   10.1.4.203
+master-user     =   repl
+master-password =   qERUG8wf
+master-connect-retry=60
+replicate-wild-do-table=traffic.daily%
+max_relay_log_size=500M
+expire_logs_days = 3
-confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt
+replicate-wild-do-table=traffic.%</pre>
-  ssh backup1 hostname
+on master:
- ssh backup2 hostname
+  touch /usr/home/working/.lock
+(make sure processflows not running)
-* edit root's path and login script:
+<pre>FLUSH TABLES WITH READ LOCK;
-  vi /root/.cshrc
+cd /usr/home/database/traffic
+tar -czf mysql-traffic-snapshot.tgz ./daily*
+(~1G)
+SHOW MASTER STATUS;
++-----------------+-----------+--------------+------------------+
+| File            | Position  | Binlog_Do_DB | Binlog_Ignore_DB |
++-----------------+-----------+--------------+------------------+
+| bwdb-bin.000039 | 154432615 |              |                  |
++-----------------+-----------+--------------+------------------+
-Change alias entries (add G):
+(write down info)
-<pre>alias la        ls -aG
+UNLOCK TABLES;
-alias lf        ls -FAG
+scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>
-alias ll        ls -lAG
-alias ls        ls -AG
-</pre>
-and alter the prompt, set the following:
+on slave:
- set prompt = "`/bin/hostname -s` %/# "
+<pre>mkdir /mnt/data1/db/mysql/traffic
+cd /mnt/data1/db/mysql/traffic/
+tar xzvf mysql-traffic-snapshot.tgz
+(restart mysql)
+CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
+START SLAVE;</pre>
-* install cvsup
+<pre>cd /usr/home/database/traffic
- cd /usr/ports/net/cvsup-without-gui
+scp *</pre>
- make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null
-* get latest sources for this release:
+<pre>optimize table dailyPortTotals_69_55_224;
-<pre>cd /usr/src
+optimize table dailyPortTotals_69_55_225;
-echo "*default host=cvsup4.freebsd.org\
+optimize table dailyPortTotals_69_55_226;
-*default base=/usr\
+optimize table dailyPortTotals_69_55_227;
-*default prefix=/usr\
+optimize table dailyPortTotals_69_55_228;
-*default release=cvs tag=RELENG_8_3\
+optimize table dailyPortTotals_69_55_229;
-*default delete use-rel-suffix\
+optimize table dailyPortTotals_69_55_230;
-*default compress\
+optimize table dailyPortTotals_69_55_231;
-src-all" > sup
+optimize table dailyPortTotals_69_55_232;
+optimize table dailyPortTotals_69_55_233;
+optimize table dailyPortTotals_69_55_234;
+optimize table dailyPortTotals_69_55_235;
+optimize table dailyPortTotals_69_55_236;
+optimize table dailyPortTotals_69_55_237;
+optimize table dailyPortTotals_69_55_238;
+optimize table dailyPortTotals_69_55_239;</pre>
-cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>
+== Build ==
-* configure new kernel
+=== BIOS Config ===
+disable quiet boot
- cd /usr/src/sys/amd64/conf
+set to last state after power loss
- scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb
-Edit config and change name:
+set date/time to GMT
- vi bwdb
- ident  bwdb
-* build, install kernel and world
+enable serial console output (baud rate 115200)
-<pre>cd /boot
+=== Install OS ===
-mv kernel kernel.GENERIC
+Install FreeBSD 8.3 amd64
-cd kernel.GENERIC
-cd /usr/src
-make buildkernel installkernel
-make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
+* partition map:
-(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
+<pre>/ 500m
-make installworld
+swap 4096m
-(2450: 3min, supermicro: 1min, 2950: :34)
+/var 256m
-mergemaster -i</pre>
+/tmp 256m
+/usr ~</pre>
-* populate /etc/rc.conf with IPs and NFS settings
+* edit /etc/make.conf
+Castle:
+<pre>echo "WITHOUT_X11=yes \
+KERNCONF=bwdb \
+BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>
-castle:
+i2b:
-<pre>vi /etc/rc.conf
+<pre>echo "WITHOUT_X11=yes \
+KERNCONF=bwdb2 \
+BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>
-hostname="bwdb.johncompanies.com"
+* add settings to /boot/loader.conf and /boot.config
-kern_securelevel_enable="NO"
-portmap_enable="NO"
-sendmail_enable="NO"
-usbd_enable="YES"
-xntpd_enable="YES"
+<pre>echo "-Dh" >> /boot.config
-nfs_client_enable="YES"
-nfs_reserved_port_only="YES"
-ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
-ifconfig_em0="up promisc"
-defaultrouter="10.1.4.1"
-snmpd_enable="YES"
-inetd_enable="YES"
+echo 'console="comconsole,vidconsole" \
-inetd_flags="-wW -a 10.1.4.203"
+boot_multicons="YES" \
-fsck_y_enable="YES"
+boot_serial="YES" \
-background_fsck="NO"
+comconsole_speed="115200"' >> /boot/loader.conf</pre>
-sshd_enable="YES"
-ipfw_load="YES"</pre>
-i2b:
+* turn off all ttyv's except 0 and 1 in /etc/ttys
-<pre>vi /etc/rc.conf
+also turn on ttyu0, change type to vt100:
+<pre>vi /etc/ttys
-hostname="bwdb2.johncompanies.com"
+ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
-kern_securelevel_enable="NO"
+ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
-portmap_enable="NO"
+ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
-sendmail_enable="NO"
+ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
-usbd_enable="YES"
+ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
+# Serial terminals
+# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
+ttyu0   "/usr/libexec/getty std.9600"   vt100   on secure
-xntpd_enable="YES"
+kill -1 1</pre>
-nfs_client_enable="YES"
-nfs_reserved_port_only="YES"
-ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
-ifconfig_em0="up promisc"
-defaultrouter="10.1.2.1"
-snmpd_enable="YES"
-inetd_enable="YES"
+on console server:
-inetd_flags="-wW -a 10.1.2.4"
+ vi /etc/remote
-fsck_y_enable="YES"
+(rename port to jail8 depending on where and which digi plugged into)
-background_fsck="NO"
+test serial console
-sshd_enable="YES"
-ipfw_load="YES"</pre>
-* reboot. Confirm new kernel is loaded
- uname -a
+* populate hosts
+i2b:
+<pre>echo "69.55.230.10 backup2" >> /etc/hosts
+echo "69.55.230.11 backup1" >> /etc/hosts
+echo "10.1.2.3 backup3" >> /etc/hosts</pre>
-* update ports:
+castle:
-<pre>cd /usr/ports
+<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
-echo "*default host=cvsup4.FreeBSD.org\
+echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
-*default base=/usr\
+echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
-*default prefix=/usr\
+</pre>
-*default release=cvs tag=RELENG_8_3\
-*default delete use-rel-suffix\
-*default compress\
-ports-all tag=." > sup
-cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>
+* put key in authorized_keys on backup1 and  backup2
+ cd
+ ssh-keygen -t dsa -b 1024
+(default location, leave password blank)
-* Install raid mgmt tool
+castle:
+ cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
+ cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
-<pre>cd /usr/local/sbin
+i2b:
-scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
+ cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
-tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
+ cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
-rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
+ cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
-chmod 0700 tw_cli</pre>
-Test:
+confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt
- ./tw_cli info c0
-Grab raid check script:
+ ssh backup1 hostname
-  scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc
+  ssh backup2 hostname
-Setup cronjob:
+* edit root's path and login script:
-<pre>crontab -e
+ vi /root/.cshrc
-*/5 * * * * /usr/local/etc/3wraidchk</pre>
-* install rsync from ports
+Change alias entries (add G):
- cd /usr/ports/net/rsync
+<pre>alias la        ls -aG
- make install clean
+alias lf        ls -FAG
+alias ll        ls -lAG
+alias ls        ls -AG
+</pre>
-choose default options
+and alter the prompt, set the following:
+ set prompt = "`/bin/hostname -s` %/# "
+* install cvsup
+ cd /usr/ports/net/cvsup-without-gui
+ make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null
-* install perl from ports
+* get latest sources for this release:
- cd /usr/ports/lang/perl5.8
+<pre>cd /usr/src
- make install clean
+echo "*default host=cvsup4.freebsd.org\
+*default base=/usr\
+*default prefix=/usr\
+*default release=cvs tag=RELENG_8_3\
+*default delete use-rel-suffix\
+*default compress\
+src-all" > sup
+cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>
-choose default options
+* configure new kernel
-* install bb client
+ cd /usr/src/sys/amd64/conf
+ scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb
-Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
+Edit config and change name:
-  rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/
+  vi bwdb
+ ident  bwdb
- adduser
+* build, install kernel and world
-Output/response:
+<pre>cd /boot
-<pre>Username: bb
+mv kernel kernel.GENERIC
-Full name: bb
+cd kernel.GENERIC
-Uid (Leave empty for default): 1984
+cd /usr/src
-Login group [bb]:
+make buildkernel installkernel
-Login group is bb. Invite bb into other groups? []:
-Login class [default]:
-Shell (sh csh tcsh nologin) [sh]:
-Home directory [/home/bb]:
-Use password-based authentication? [yes]:
-Use an empty password? (yes/no) [no]:
-Use a random password? (yes/no) [no]: yes
-Lock out the account after creation? [no]:
-Username   : bb
-Password   : <random>
-Full Name  : bb
-Uid        : 1984
-Class      :
-Groups     : bb
-Home       : /home/bb
-Shell      : /bin/sh
-Locked     : no
-OK? (yes/no): yes</pre>
- cd /usr/home/bb
+make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
- scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
+(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
- tar xzf bb-freebsd_linuxcompat.tgz
+make installworld
+(2450: 3min, supermicro: 1min, 2950: :34)
+mergemaster -i</pre>
-edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
+* populate /etc/rc.conf with IPs and NFS settings
- echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
+castle:
-.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
+<pre>vi /etc/rc.conf
-Edit for machine name and private IP.
+hostname="bwdb.johncompanies.com"
+kern_securelevel_enable="NO"
+portmap_enable="NO"
+sendmail_enable="NO"
+usbd_enable="YES"
-if this machine is at i2b:
+xntpd_enable="YES"
- echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
+nfs_client_enable="YES"
-.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
+nfs_reserved_port_only="YES"
+ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
+ifconfig_em0="up promisc"
+defaultrouter="10.1.4.1"
+snmpd_enable="YES"
-<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles
+inetd_enable="YES"
+inetd_flags="-wW -a 10.1.4.203"
+fsck_y_enable="YES"
+background_fsck="NO"
+sshd_enable="YES"
+ipfw_load="YES"</pre>
+i2b:
+<pre>vi /etc/rc.conf
+hostname="bwdb2.johncompanies.com"
+kern_securelevel_enable="NO"
+portmap_enable="NO"
+sendmail_enable="NO"
+usbd_enable="YES"
-MACHINE="bwdb,johncompanies,com"      # HAS TO BE IN A,B,C FORM</pre>
+xntpd_enable="YES"
-Edit for machine name.
+nfs_client_enable="YES"
+nfs_reserved_port_only="YES"
+ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
+ifconfig_em0="up promisc"
+defaultrouter="10.1.2.1"
+snmpd_enable="YES"
-Have bb watch for flow-capture, mysql
+inetd_enable="YES"
-<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
+inetd_flags="-wW -a 10.1.2.4"
-localhost: flow-capture :
+fsck_y_enable="YES"
-localhost: mysqld :</pre>
+background_fsck="NO"
+sshd_enable="YES"
+ipfw_load="YES"</pre>
+* reboot. Confirm new kernel is loaded
-<pre>cd /usr/home/bb/bbc1.9e-btf/etc
+ uname -a
-./bbchkcfg.sh
-(y to questions)
-./bbchkhosts.sh
-(ignore ssh errors)
-cd ../..
-chown -R bb .
-su bb
-cd
-cd bbc1.9e-btf
-./runbb.sh start
-more BBOUT
-(look for errors)
-exit</pre>
-Put in script to start bb @ boot:
+* update ports:
- echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
+<pre>cd /usr/ports
- chmod +x /usr/local/etc/rc.d/bb.sh
+echo "*default host=cvsup4.FreeBSD.org\
+*default base=/usr\
+*default prefix=/usr\
+*default release=cvs tag=RELENG_8_3\
+*default delete use-rel-suffix\
+*default compress\
+ports-all tag=." > sup
+cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>
-If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:
+* Install raid mgmt tool
-ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2
+<pre>cd /usr/local/sbin
+scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
+tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
+rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
+chmod 0700 tw_cli</pre>
+Test:
+ ./tw_cli info c0
-* configure bb on mail
+Grab raid check script:
-<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
+ scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc
-.1.4.203 bwdb.johncompanies.com # ssh
-su bb
+Setup cronjob:
-cd
+<pre>crontab -e
-bbsrc/bb/runbb.sh restart ; exit</pre>
+*/5 * * * * /usr/local/etc/3wraidchk</pre>
-* configure ntp server
+* install rsync from ports
-Castle:
+  cd /usr/ports/net/rsync
-  echo "server 10.1.4.1" > /etc/ntp.conf
+ make install clean
-I2b:
+choose default options
- echo "server 10.1.2.1" > /etc/ntp.conf
-<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
+* install perl from ports
-sleep 2; ntpq -p</pre>
+ cd /usr/ports/lang/perl5.8
-(confirm it’s able to reach our time server)
+ make install clean
-<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
+choose default options
-chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>
-* fwd and reverse lookups on ns1c
+* install bb client
-vr johncompanies.com
- (edit the PTR too)
-* setup backups
+Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
-<pre>echo '#\!/bin/sh\
+ rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/
-backupdir=/data/bwdb/current\
-server=backup1\
-\
-## ENTRY /etc\
-## ENTRY /usr/home/flowbin\
-## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>
-Castle:
+  adduser
-setup backup dirs:
-  ssh backup1 mkdir -p /data/bwdb/current
-on backup1, add the system to
- vi /usr/local/sbin/snapshot_rotate
-I2b:
+Output/response:
-setup backup dirs:
- ssh backup3 mkdir -p /data/bwdb/current
-on backup3, add the system to
- vi /usr/local/sbin/snapshot_archive
+<pre>Username: bb
+Full name: bb
+Uid (Leave empty for default): 1984
+Login group [bb]:
+Login group is bb. Invite bb into other groups? []:
+Login class [default]:
+Shell (sh csh tcsh nologin) [sh]:
+Home directory [/home/bb]:
+Use password-based authentication? [yes]:
+Use an empty password? (yes/no) [no]:
+Use a random password? (yes/no) [no]: yes
+Lock out the account after creation? [no]:
+Username   : bb
+Password   : <random>
+Full Name  : bb
+Uid        : 1984
+Class      :
+Groups     : bb
+Home       : /home/bb
+Shell      : /bin/sh
+Locked     : no
+OK? (yes/no): yes</pre>
-Copy over the backup script:
+ cd /usr/home/bb
-  scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/
+  scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
+ tar xzf bb-freebsd_linuxcompat.tgz
-Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>
-<pre>crontab -e
+edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
-0 * * * /usr/local/etc/rsync.backup</pre>
-* make /root/logs
+ echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
- mkdir /root/logs
+.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
-* edit sshd_config for security
+Edit for machine name and private IP.
-<pre>vi /etc/ssh/sshd_config
+if this machine is at i2b:
-ListenAddress 10.1.4.203
+ echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
-PermitRootLogin yes
+.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
-kill -1 `cat /var/run/sshd.pid`</pre>
+<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles
-Edit for private IP.
+MACHINE="bwdb,johncompanies,com"      # HAS TO BE IN A,B,C FORM</pre>
+Edit for machine name.
-* snmp
+Have bb watch for flow-capture, mysql
+<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
+localhost: flow-capture :
+localhost: mysqld :</pre>
-(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
-<pre>
-cd /usr/ports/net-mgmt/net-snmp
-make install clean
-(defaults)
-cat >> /etc/rc.conf
+<pre>cd /usr/home/bb/bbc1.9e-btf/etc
-snmpd_enable="YES"
+./bbchkcfg.sh
-snmpd_flags="-a"
+(y to questions)
-snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
+./bbchkhosts.sh
-snmptrapd_enable="YES"
+(ignore ssh errors)
-snmptrapd_flags="-a -p /var/run/snmptrapd.pid"
+cd ../..
+chown -R bb .
+su bb
+cd
+cd bbc1.9e-btf
+./runbb.sh start
+more BBOUT
+(look for errors)
+exit</pre>
-cat > /usr/local/share/snmp/snmpd.conf
+Put in script to start bb @ boot:
-rocommunity  jcread 10.1.4.5
+ echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
-rocommunity  jcread 10.1.4.202
+  chmod +x /usr/local/etc/rc.d/bb.sh
-</pre>
-=== netflow ===
-Install flow tools:
+If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:
-<pre>cd /usr/ports/net-mgmt/flow-tools
-make install clean</pre>
-Defaults.
- mkdir /usr/home/flows
+ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2
-Flow start script:
- echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh
- chmod 0700 /usr/local/etc/rc.d/flow-capture.sh
+* configure bb on mail
+<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
+.1.4.203 bwdb.johncompanies.com # ssh
+su bb
+cd
+bbsrc/bb/runbb.sh restart ; exit</pre>
-Edit for private IP.
+* configure ntp server
+Castle:
+ echo "server 10.1.4.1" > /etc/ntp.conf
-Netgraph start script:
+I2b:
-<pre>
+ echo "server 10.1.2.1" > /etc/ntp.conf
-cat > /usr/local/etc/rc.d/netgraph.sh
-/usr/sbin/ngctl -f- <<-SEQ
+<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
-mkpeer em0: netflow lower iface0
+sleep 2; ntpq -p</pre>
-name em0:lower netflow
+(confirm it’s able to reach our time server)
-connect em0: netflow: upper out0
-mkpeer netflow: ksocket export inet/dgram/udp
-msg netflow:export connect inet/10.1.4.203:4444
-SEQ
-#/usr/sbin/ngctl -f- <<-SEQ
+<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
-#shutdown netflow:
+chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>
-#SEQ
-chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
+* fwd and reverse lookups on ns1c
-Edit for private IP.
+vr johncompanies.com
+ (edit the PTR too)
-Confirm netflow is running after running scripts:
+* setup backups
-<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
+<pre>echo '#\!/bin/sh\
-Available commands:
+backupdir=/data/bwdb/current\
-  config     get or set configuration of node at <path>
+server=backup1\
-  connect    Connects hook <peerhook> of the node at <relpath> to <hook>
+\
-  debug      Get/set debugging verbosity level
+## ENTRY /etc\
-  dot        Produce a GraphViz (.dot) of the entire netgraph.
+## ENTRY /usr/home/flowbin\
-  help       Show command summary or get more help on a specific command
+## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>
-  list       Show information about all nodes
-  mkpeer     Create and connect a new node to the node at "path"
-  msg        Send a netgraph control message to the node at "path"
-  name       Assign name <name> to the node at <path>
-  read       Read and execute commands from a file
-  rmhook     Disconnect hook "hook" of the node at "path"
-  show       Show information about the node at <path>
-  shutdown   Shutdown the node at <path>
-  status     Get human readable status information from the node at <path>
-  types      Show information about all installed node types
-  write      Send a data packet down the hook named by "hook".
-  quit       Exit program
-+ show netflow:
-  Name: netflow         Type: netflow         ID: 00000004   Num hooks: 3
-  Local hook      Peer name       Peer type    Peer ID         Peer hook
-  ----------      ---------       ---------    -------         ---------
-  export          <unnamed>       ksocket      00000005        inet/dgram/udp
-  out0            em0             ether        00000001        upper
-  iface0          em0             ether        00000001        lower
-+
-</pre>
-We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):
+Castle:
+setup backup dirs:
+ ssh backup1 mkdir -p /data/bwdb/current
+on backup1, add the system to
+ vi /usr/local/sbin/snapshot_rotate
-<pre>
+I2b:
-crontab -e
+setup backup dirs:
-#restart flow-capture
+ ssh backup3 mkdir -p /data/bwdb/current
-*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
+on backup3, add the system to
-</pre>
+ vi /usr/local/sbin/snapshot_archive
-==== process flow tools ====
-<pre>mkdir /usr/home/flowbin
+Copy over the backup script:
-mkdir /usr/home/working</pre>
+ scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/
-Install modules:
+Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>
-<pre>cd /usr/ports/devel/p5-Date-Calc
-make install clean
-cd /usr/ports/mail/p5-Mail-Sendmail
-make install clean</pre>
-Queue script:
+<pre>crontab -e
-<pre>
+0 * * * /usr/local/etc/rsync.backup</pre>
-cat > /usr/home/flowbin/queue.pl
-#!/usr/bin/perl
-use strict;
+* make /root/logs
+ mkdir /root/logs
-BEGIN {
+* edit sshd_config for security
-    push @INC, "/usr/home/flowbin";
-}
-use date;
+<pre>vi /etc/ssh/sshd_config
+ListenAddress 10.1.4.203
+PermitRootLogin yes
-my $flowbase = "/usr/home/flows";
+kill -1 `cat /var/run/sshd.pid`</pre>
-#my $flowqueue = "/usr/home/queue";
-my $flowqueue = "/usr/home/working";
-my ($date, $time) = date::CurrentDateTime();
+Edit for private IP.
-my $flowdir = mkFlowDir($date);
+* snmp
-`mv $flowdir/ft-* $flowqueue`;
-if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
+(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
-    my $newdate = date::AddDays($date, -1);
+<pre>
-    my $flowdir = mkFlowDir($newdate);
+cd /usr/ports/net-mgmt/net-snmp
-    `mv $flowdir/ft-* $flowqueue`;
+make install clean
-}
+(defaults)
+cat >> /etc/rc.conf
+snmpd_enable="YES"
+snmpd_flags="-a"
+snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
+snmptrapd_enable="YES"
+snmptrapd_flags="-a -p /var/run/snmptrapd.pid"
-sub mkFlowDir {
+cat > /usr/local/share/snmp/snmpd.conf
-    my $date = shift;
+rocommunity  jcread 10.1.4.5
-    $date =~ /([0-9]{4}-[0-9]{2})/;
+rocommunity  jcread 10.1.4.202
-    my $yearmonth = $1;
-    return "$flowbase/$yearmonth/$date";
-}
 </pre>
-Date.pm module:
+=== netflow ===
-<pre>
-cat > /usr/home/flowbin/date.pm
+Install flow tools:
-#!/usr/local/bin/perl
+<pre>cd /usr/ports/net-mgmt/flow-tools
-#
+make install clean</pre>
-# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
+Defaults.
-#
-# Copyright (c) 2001, 2002, 2003
-#      e-Monitoring Networks, Inc.  All rights reserved.
-#
-#
-#
-# date.pl - Higher level functions written on top of Date::Calc
-package date;
+ mkdir /usr/home/flows
-use strict;
+Flow start script:
-use Date::Calc qw(:all);
+ echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh
-sub DayDiff { #calculate the difference in days from two dates
+ chmod 0700 /usr/local/etc/rc.d/flow-capture.sh
-    my $date1 = shift;
-    my $date2 = shift;
-    my ($year1, $month1, $day1) = &DateToymd($date1);
-    my ($year2, $month2, $day2) = &DateToymd($date2);
-    my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
-    return $diff;
-}
-sub AddDays { #adds specified number of days to the supplied date
+Edit for private IP.
-    my $date = shift;
-    my $days = shift;
-    my ($year, $month, $day) = &DateToymd($date);
-    my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
-    my $ndate = &ymdToDate($nyear, $nmonth, $nday);
-    return $ndate;
-}
-sub AddHours { #adds specified number of hours to the supplied date and time
+Netgraph start script:
-    my $date = shift;
+<pre>
-    my $time = shift;
+cat > /usr/local/etc/rc.d/netgraph.sh
-    my $addhours = shift;
-    my $adddays = 0;
-    if (abs($addhours / 24) >= 1) {
-        $adddays = int($addhours / 24);
-        $addhours -= $adddays * 24;
-    }
-    my ($year, $month, $day) = &DateToymd($date);
-    my ($hour, $minute, $second) = &TimeTohms($time);
-    my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
-                                                          $hour, $minute, $second,
-                                                          $adddays, $addhours, 0, 0);
-    my $ndate = &ymdToDate($ny, $nm, $nd);
-    my $ntime = &hmsToTime($nh, $nmin, $ns);
-    return $ndate, $ntime;
-}
-sub AddMinutes {
+/usr/sbin/ngctl -f- <<-SEQ
-    my $date = shift;
+mkpeer em0: netflow lower iface0
-    my $time = shift;
+name em0:lower netflow
-    my $minutes = shift;
+connect em0: netflow: upper out0
-    my ($year, $month, $day) = &DateToymd($date);
+mkpeer netflow: ksocket export inet/dgram/udp
-    my ($hour, $minute, $second) = &TimeTohms($time);
+msg netflow:export connect inet/10.1.4.203:4444
-    my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
+SEQ
-                                                          $hour, $minute, $second,
-, 0, $minutes, 0);
-    my $ndate = &ymdToDate($ny, $nm, $nd);
-    my $ntime = &hmsToTime($nh, $nmin, $ns);
-    return $ndate, $ntime;
-}
-sub CurrentDateTime { #return the current date and time
+#/usr/sbin/ngctl -f- <<-SEQ
-    my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
+#shutdown netflow:
-    my $date = &ymdToDate($y, $m, $d);
+#SEQ
-    my $time = &hmsToTime($h, $min, $s);
-    return $date, $time;
-}
-sub Currentymd { #return the current year, month and day as separate variables
+chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
-    my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
+Edit for private IP.
-    return $y, $m, $d;
-}
-sub DateToymd { #takes a date and returns year, month, day as individual values
+Confirm netflow is running after running scripts:
-     my $date = shift;
+<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
-     if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
+Available commands:
-        my $day = $3;
+  config     get or set configuration of node at <path>
-        my $month = $2;
+  connect    Connects hook <peerhook> of the node at <relpath> to <hook>
-        my $year = $1;
+  debug      Get/set debugging verbosity level
-        return $year, $month, $day;
+  dot        Produce a GraphViz (.dot) of the entire netgraph.
-    }
+  help       Show command summary or get more help on a specific command
-    return undef;
+  list       Show information about all nodes
-}
+  mkpeer     Create and connect a new node to the node at "path"
+  msg        Send a netgraph control message to the node at "path"
+  name       Assign name <name> to the node at <path>
+  read       Read and execute commands from a file
+  rmhook     Disconnect hook "hook" of the node at "path"
+  show       Show information about the node at <path>
+  shutdown   Shutdown the node at <path>
+  status     Get human readable status information from the node at <path>
+  types      Show information about all installed node types
+  write      Send a data packet down the hook named by "hook".
+  quit       Exit program
++ show netflow:
+  Name: netflow         Type: netflow         ID: 00000004   Num hooks: 3
+  Local hook      Peer name       Peer type    Peer ID         Peer hook
+  ----------      ---------       ---------    -------         ---------
+  export          <unnamed>       ksocket      00000005        inet/dgram/udp
+  out0            em0             ether        00000001        upper
+  iface0          em0             ether        00000001        lower
++
+</pre>
+We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):
-sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
+<pre>
-    my $time = shift;
+crontab -e
-    if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
+#restart flow-capture
-        my $hour = $1;
+*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
-        my $minute = $2;
+</pre>
-        my $second = $3;
-        if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
-        if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
-        if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
-        return $hour, $minute, $second;
-    }
-    return undef;
-}
-sub ymdToDate { #takes year, month, day and assembles them into our date format
+==== process flow tools ====
-    my $year = shift;
-    my $month = shift;
+<pre>mkdir /usr/home/flowbin
-    my $day = shift;
+mkdir /usr/home/working</pre>
-    if (defined($year) && defined($month) && defined ($day)) {
-        $month = sprintf("%02d", $month);
-        $day = sprintf("%02d", $day);
-        return "$year-$month-$day";
-    }
-    return undef;
-}
-sub hmsToTime { #takes hour minute and second and assembles them into our time format
+Install modules:
-    my $hour = shift;
+<pre>cd /usr/ports/devel/p5-Date-Calc
-    my $minute = shift;
+make install clean
-    my $second = shift;
+cd /usr/ports/mail/p5-Mail-Sendmail
-    if (defined($hour) && defined($minute) && defined ($second)) {
+make install clean</pre>
-        if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
-        if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
-        if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
-        return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
-    }
-    return undef;
-}
-sub CompareDates { #compares two date and time pairs
+Queue script:
-    my $date1 = shift;
+<pre>
-    my $time1 = shift;
+cat > /usr/home/flowbin/queue.pl
-    my $date2 = shift;
+#!/usr/bin/perl
-    my $time2 = shift;
-    my ($year1, $month1, $day1) = &DateToymd($date1);
+use strict;
-    my ($hour1, $minute1, $second1) = &TimeTohms($time1);
-    my ($year2, $month2, $day2) = &DateToymd($date2);
-    my ($hour2, $minute2, $second2) = &TimeTohms($time2);
-#    &debug("$year1, $month1, $day1, $year2, $month2, $day2");
+BEGIN {
-    my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
+     push @INC, "/usr/home/flowbin";
-    if ($days > 0) { return 1;}
+}
-    if ($days < 0) { return -1;}
-     if ($days == 0) { #same day, compare times
+use date;
-        my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
-        my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
-        if ($seconds1 < $seconds2) { return 1;}
-        if ($seconds1 > $seconds2) { return -1;}
-        if ($seconds1 == $seconds2) { return 0;}
-    }
-    return undef;
-}
-sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
+my $flowbase = "/usr/home/flows";
-    my $date1 = shift;
+#my $flowqueue = "/usr/home/queue";
-    my $time1 = shift;
+my $flowqueue = "/usr/home/working";
-    my $date2 = shift;
-    my $time2 = shift;
-    my $window = shift;
-    my ($year1, $month1, $day1) = &DateToymd($date1);
+my ($date, $time) = date::CurrentDateTime();
-    my ($hour1, $minute1, $second1) = &TimeTohms($time1);
-    my ($year2, $month2, $day2) = &DateToymd($date2);
-    my ($hour2, $minute2, $second2) = &TimeTohms($time2);
-    my ($day, $hour, $minute, $second) =
+my $flowdir = mkFlowDir($date);
-        &Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
+`mv $flowdir/ft-* $flowqueue`;
-                    $year2, $month2, $day2, $hour2, $minute2, $second2);
-    $minute *= 60;
-    $hour *= 3600;
-    $day *= 86400;
-    my $total = $second + $minute + $hour + $day;
-    if (abs($total) < $window) {
-        return 1;
-    }
-    return 0;
-}
-sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
+if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
-    my $date1 = shift;
+     my $newdate = date::AddDays($date, -1);
-    my $time1 = shift;
+     my $flowdir = mkFlowDir($newdate);
-    my $date2 = shift;
+     `mv $flowdir/ft-* $flowqueue`;
-    my $time2 = shift;
-    my $date3 = shift;
-    my $time3 = shift;
-    if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
-        return 0;
-     }
-    if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
-        return 0;
-     }
-     return 1;
 }
-sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
+sub mkFlowDir {
      my $date = shift;
-     my $time = shift;
+     $date =~ /([0-9]{4}-[0-9]{2})/;
-    my ($year, $month, $day) = &DateToymd($date);
+     my $yearmonth = $1;
-    my ($hour, $minute, $second) = &TimeTohms($time);
+     return "$flowbase/$yearmonth/$date";
-    my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
-                                      $year, $month, $day, $hour, $minute, $second);
-     my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
-     return $seconds;
 }
+</pre>
-sub SecondsToDateTime { #converts seconds since epoch to date/time
+Date.pm module:
-    my $seconds = shift;
+<pre>
-    my $days = int($seconds / 86400);
+cat > /usr/home/flowbin/date.pm
-    $seconds -= $days * 86400;
+#!/usr/local/bin/perl
-    my $hours = int($seconds / 3600);
+#
-    $seconds -= $hours * 3600;
+# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
-    my $minutes = int($seconds / 60);
+#
-    $seconds -= $minutes * 60;
+# Copyright (c) 2001, 2002, 2003
-    my ($year, $month, $day, $hour, $minute, $second) =
+#      e-Monitoring Networks, Inc.  All rights reserved.
-        &Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
+#
-    $month = sprintf("%02d", $month);
+#
-    $day = sprintf("%02d", $day);
+#
-    $hour = sprintf("%02d", $hour);
+# date.pl - Higher level functions written on top of Date::Calc
-    $minute = sprintf("%02d", $minute);
-    $second = sprintf("%02d", $second);
-    return "$year-$month-$day", "$hour:$minute:$second";
-}
-sub DateToDayName {
-    my $date = shift;
-    my ($year, $month, $day) = &DateToymd($date);
-    my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
-    $name =~ /^[A-Za-z]{3}/;
-    $name = $&;
-    return $name;
-}
-sub ValiDate {
-    return @_;
+package date;
+use strict;
+use Date::Calc qw(:all);
+sub DayDiff { #calculate the difference in days from two dates
+    my $date1 = shift;
+    my $date2 = shift;
+    my ($year1, $month1, $day1) = &DateToymd($date1);
+    my ($year2, $month2, $day2) = &DateToymd($date2);
+    my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
+    return $diff;
 }
-sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
+sub AddDays { #adds specified number of days to the supplied date
      my $date = shift;
+    my $days = shift;
      my ($year, $month, $day) = &DateToymd($date);
-     if (Day_of_Week($year,$month,$day) < 6) { return 1; }
+     my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
-     else { return 0; }
+    my $ndate = &ymdToDate($nyear, $nmonth, $nday);
+     return $ndate;
 }
-; #don't remove this line
+sub AddHours { #adds specified number of hours to the supplied date and time
-</pre>
+    my $date = shift;
+    my $time = shift;
+    my $addhours = shift;
+    my $adddays = 0;
+    if (abs($addhours / 24) >= 1) {
+        $adddays = int($addhours / 24);
+        $addhours -= $adddays * 24;
+    }
+    my ($year, $month, $day) = &DateToymd($date);
+    my ($hour, $minute, $second) = &TimeTohms($time);
+    my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
+                                                          $hour, $minute, $second,
+                                                          $adddays, $addhours, 0, 0);
+    my $ndate = &ymdToDate($ny, $nm, $nd);
+    my $ntime = &hmsToTime($nh, $nmin, $ns);
+    return $ndate, $ntime;
+}
- chmod 0700 /usr/home/flowbin/queue.pl
+sub AddMinutes {
+    my $date = shift;
+    my $time = shift;
+    my $minutes = shift;
+    my ($year, $month, $day) = &DateToymd($date);
+    my ($hour, $minute, $second) = &TimeTohms($time);
+    my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
+                                                          $hour, $minute, $second,
+, 0, $minutes, 0);
+    my $ndate = &ymdToDate($ny, $nm, $nd);
+    my $ntime = &hmsToTime($nh, $nmin, $ns);
+    return $ndate, $ntime;
+}
-Setup cronjob:
+sub CurrentDateTime { #return the current date and time
-<pre>crontab -e
+    my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
-#move flow data into the queue
+    my $date = &ymdToDate($y, $m, $d);
-,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>
+    my $time = &hmsToTime($h, $min, $s);
+    return $date, $time;
+}
-==== flow processing: i2b ====
+sub Currentymd { #return the current year, month and day as separate variables
-<pre>cat > /usr/home/flowbin/processflows-sql.pl
+    my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
-#!/usr/bin/perl
+    return $y, $m, $d;
+}
-#use strict;
+sub DateToymd { #takes a date and returns year, month, day as individual values
-#$debug=1;
+    my $date = shift;
-#$dry=1;
+    if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
+        my $day = $3;
+        my $month = $2;
+        my $year = $1;
+        return $year, $month, $day;
+    }
+    return undef;
+}
-my $log = '/usr/home/flowbin/discards.log';
+sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
+    my $time = shift;
+    if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
+        my $hour = $1;
+        my $minute = $2;
+        my $second = $3;
+        if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
+        if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
+        if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
+        return $hour, $minute, $second;
+    }
+    return undef;
+}
-use Data::Dumper;
+sub ymdToDate { #takes year, month, day and assembles them into our date format
+    my $year = shift;
-BEGIN {
+    my $month = shift;
-    push @INC, "/usr/home/flowbin";
+    my $day = shift;
+    if (defined($year) && defined($month) && defined ($day)) {
+        $month = sprintf("%02d", $month);
+        $day = sprintf("%02d", $day);
+        return "$year-$month-$day";
+    }
+    return undef;
 }
-#my $queuedir = "/usr/home/queue";
+sub hmsToTime { #takes hour minute and second and assembles them into our time format
-my $queuedir = "/usr/home/working";
+    my $hour = shift;
-my $archivedir = "/usr/home/archive";
+    my $minute = shift;
-my $sqldir = "/usr/home/sql";
+    my $second = shift;
-my $sqldirworking = "/usr/home/sql/tmp";
+    if (defined($hour) && defined($minute) && defined ($second)) {
+         if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
-unless ($dry) {
+         if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
-    if (-e "$queuedir/.lock") {
+         if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
-         open(FILE, "$queuedir/.lock");
+         return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
-        my $pid = <FILE>;
-         chomp($pid);
-        close(FILE);
-         if (kill(0, $pid)) {
-            #another process is using the queue, bail out
-            exit(0);
-        }
-         else {
-            #dead lock file, remove it
-            `rm $queuedir/.lock`;
-        }
      }
-     open(FILE, "> $queuedir/.lock");
+     return undef;
-    print FILE "$$\n";
-    close(FILE);
 }
-opendir(DIR, $queuedir);
+sub CompareDates { #compares two date and time pairs
-my @files = readdir(DIR);
+    my $date1 = shift;
-closedir(DIR);
+    my $time1 = shift;
+    my $date2 = shift;
+    my $time2 = shift;
-foreach my $file (sort @files) {
+    my ($year1, $month1, $day1) = &DateToymd($date1);
-    unless($file =~ /^\./) {
+    my ($hour1, $minute1, $second1) = &TimeTohms($time1);
-        $file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
+    my ($year2, $month2, $day2) = &DateToymd($date2);
-        my $date = "$1 $2:$3:$4";
+    my ($hour2, $minute2, $second2) = &TimeTohms($time2);
-        my $outfile = "$1-$2:$3.sql";
-        unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
-        my $condensedDate = $1;
-        $condensedDate =~ s/-//g;
-        my $iptotal = {};
-        my $protototal = {};
-        my $porttotal = {};
-        &debug("started file $file at ");
+#    &debug("$year1, $month1, $day1, $year2, $month2, $day2");
-        &debug(`date`);
+    my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
-        &debug("getting raw flow data (flow-print)");
+    if ($days > 0) { return 1;}
-         `cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
+    if ($days < 0) { return -1;}
-         &debug("aggregating data at ");
+    if ($days == 0) { #same day, compare times
-        &debug(`date`);
+         my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
-         unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
+         my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
-         LOOP: while (my $line = readline DATA) {
+         if ($seconds1 < $seconds2) { return 1;}
-            my @d = split /[\s]+/, $line;
+         if ($seconds1 > $seconds2) { return -1;}
-            if ($d[0] ne '' && $d[0] ne 'Start') {
+        if ($seconds1 == $seconds2) { return 0;}
-                my $addr = 0;
+    }
-                my $port = 0;
+    return undef;
+}
-                #Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
+sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
-                #0     1   2   3            4    5   6            7    8 9  10   11
+    my $date1 = shift;
-                          #|
+     my $time1 = shift;
-                          # outbound = 2, inbound = 1
+    my $date2 = shift;
+    my $time2 = shift;
+    my $window = shift;
-                my (@src_ip) = split '\.', $d[3];
+    my ($year1, $month1, $day1) = &DateToymd($date1);
-                my (@dst_ip) = split '\.', $d[6];
+    my ($hour1, $minute1, $second1) = &TimeTohms($time1);
+    my ($year2, $month2, $day2) = &DateToymd($date2);
+    my ($hour2, $minute2, $second2) = &TimeTohms($time2);
-                if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
+    my ($day, $hour, $minute, $second) =
-                   $d[2] = 2;
+        &Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
-                   # hack for outbound bulk traffic counted 2x
+                    $year2, $month2, $day2, $hour2, $minute2, $second2);
-                   #if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
+    $minute *= 60;
-                }
+    $hour *= 3600;
-                # note- this is where we filter out IPs only found at i2b
+    $day *= 86400;
-                elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
+    my $total = $second + $minute + $hour + $day;
-                   $d[2] = 1;
+    if (abs($total) < $window) {
-                }
+        return 1;
-                else { next LOOP; }
+    }
+    return 0;
+}
-                if ($d[2] == 2) {
+sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
-                    $addr = $d[3];
+    my $date1 = shift;
-                    # if the dst-port is low, store that
+    my $time1 = shift;
-                    if ($d[7] <= 1024) { $port = $d[7]; }
+    my $date2 = shift;
-                    # if the src-port is low, store that
+    my $time2 = shift;
-                    elsif ($d[4] <= 1024) { $port = $d[4]; }
+    my $date3 = shift;
-                    else { $port = 99999; }
+    my $time3 = shift;
-                }
+    if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
-                elsif ($d[2] == 1) {
+        return 0;
-                    $addr = $d[6];
+    }
-                    # if the dst-port is high, assume its return traffic, try to store src-port if low
+    if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
-                    if ($d[7] > 1024) {
+        return 0;
-                        if ($d[4] <= 1024) { $port = $d[4]; }
+    }
-                        else { $port = 99999; }
+    return 1;
-                    } else {
+}
-                        $port = $d[7];
-                    }
+sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
-                } else {
+    my $date = shift;
-                    next LOOP;
+    my $time = shift;
-                }
+    my ($year, $month, $day) = &DateToymd($date);
+    my ($hour, $minute, $second) = &TimeTohms($time);
+    my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
+                                      $year, $month, $day, $hour, $minute, $second);
+    my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
+    return $seconds;
+}
+sub SecondsToDateTime { #converts seconds since epoch to date/time
+    my $seconds = shift;
+    my $days = int($seconds / 86400);
+    $seconds -= $days * 86400;
+    my $hours = int($seconds / 3600);
+    $seconds -= $hours * 3600;
+    my $minutes = int($seconds / 60);
+    $seconds -= $minutes * 60;
+    my ($year, $month, $day, $hour, $minute, $second) =
+        &Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
+    $month = sprintf("%02d", $month);
+    $day = sprintf("%02d", $day);
+    $hour = sprintf("%02d", $hour);
+    $minute = sprintf("%02d", $minute);
+    $second = sprintf("%02d", $second);
+    return "$year-$month-$day", "$hour:$minute:$second";
+}
+sub DateToDayName {
+    my $date = shift;
+    my ($year, $month, $day) = &DateToymd($date);
+    my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
+    $name =~ /^[A-Za-z]{3}/;
+    $name = $&;
+    return $name;
+}
+sub ValiDate {
+    return @_;
-                my (@ip) = split '\.', $addr;
-                unless ($ip[0] == 69) { next LOOP; }
-                unless ($ip[1] == 55) { next LOOP; }
-                unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }
-                my $classC = "$ip[0]_$ip[1]_$ip[2]";
-#                          IP        dir
-#                if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
-#                if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
-#
-#                if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
-#                if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
-#
-#                if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
-#                if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
-                $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
-                $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];
-                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
+}
-                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];
-                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
+sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
-                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
+    my $date = shift;
-            }
+    my ($year, $month, $day) = &DateToymd($date);
-        }
+    if (Day_of_Week($year,$month,$day) < 6) { return 1; }
-        close(DATA);
+    else { return 0; }
-        `rm /usr/home/working/tmp-$file`;
+}
-        &debug("processing ip totals at ");
-        &debug(`date`);
+; #don't remove this line
-        foreach my $classC (keys(%{$iptotal})) {
+</pre>
-            my @values;
-            foreach my $ip (keys(%{$iptotal->{$classC}})) {
-                foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
-                    my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
-                    my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
-#                    $packets = $packets > 2147483647 ? 0 : $packets;
-                    if ($octets > 2147483647) {
-                        my $ddir = $dir==1 ? 'in' : 'out';
-                        #print SQL "$date $ip $ddir $octets\n";
-#                        $octets = 0;
-                    }
-                    # dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
-                    my $id = "$ip-$condensedDate-$dir";
-                    $id =~ s/\.//g;
-                    push @values, "('$date', '$ip', $dir, $octets, $packets)";
-                    my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
-                    print "$sql\n" if $dry;
-                    print SQL "$sql;\n";
-#                    $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
-                }
-            }
-            # break inserts into 100 records at a time
+ chmod 0700 /usr/home/flowbin/queue.pl
-            &debug("inserting $#values +1 values");
-            while ($#values > 0) {
-                my $sql = "insert into ipTotals_$classC values ";
-                my $max_index = $#values > 100 ? 100 : $#values;
-                for (my $i=0; $i<=$max_index; $i++) {
-                    $sql .= shift @values;
-                    $sql .= ',';
-                }
-                chop $sql;
-                print "$sql\n" if $dry;
-                print SQL "$sql;\n";
-            }
-        }
-#        &debug("processing protocol totals at ");
+Setup cronjob:
-#        &debug(`date`);
+<pre>crontab -e
-#        foreach my $classC (keys(%{$protototal})) {
+#move flow data into the queue
-#            $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
+,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>
-#            my @values;
-#            foreach my $ip (keys(%{$protototal->{$classC}})) {
+==== flow processing: i2b ====
-#                foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
+<pre>cat > /usr/home/flowbin/processflows-sql.pl
-#                    foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
+#!/usr/bin/perl
-#                        my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
-#                        my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
-# #                        $octets = $octets > 2147483647 ? 0 : $octets;
-# #                        $packets = $packets > 2147483647 ? 0 : $packets;
-#                        # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
-#                        my $id = "$ip-$condensedDate-$dir-$proto";
-#                        $id =~ s/\.//g;
-#                        push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
-#                        my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
-#                        print "$sql\n" if $dry;
-#                        $db->query($sql) unless $dry;
-# #                        $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
-#                    }
-#                }
-#            }
-#            $db->query("unlock tables") unless $dry;
-#            my $sql = "insert into protoTotals_$classC values ";
-#            $sql .= join ',', @values;
-#            $db->query("lock tables protoTotals_$classC write") unless $dry;
-#            print "$sql\n" if $dry;
-#            $db->query($sql) unless $dry;
-#            $db->query("unlock tables") unless $dry;
-#        }
-        &debug("processing port totals at ");
+#use strict;
-        &debug(`date`);
+#$debug=1;
-        foreach my $classC (keys(%{$porttotal})) {
+#$dry=1;
-            my @values;
-            foreach my $ip (keys(%{$porttotal->{$classC}})) {
-                foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
-                    foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
-                        foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
-                            my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
-                            my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
-    #                        $octets = $octets > 2147483647 ? 0 : $octets;
-    #                        $packets = $packets > 2147483647 ? 0 : $packets;
-                            # dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
+my $log = '/usr/home/flowbin/discards.log';
-                            my $id = "$ip-$condensedDate-$dir-$proto-$port";
-                            $id =~ s/\.//g;
-                            push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
-                            my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
-                            print "$sql\n" if $dry;
-                            print SQL "$sql;\n";
-    #                        $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
+use Data::Dumper;
-                        }
-                    }
-                }
-            }
-            # break inserts into 100 records at a time
+BEGIN {
-            &debug("inserting $#values +1 values");
+    push @INC, "/usr/home/flowbin";
-            while ($#values > 0) {
+}
-                my $sql = "insert into portTotals_$classC values ";
-                my $max_index = $#values > 100 ? 100 : $#values;
-                for (my $i=0; $i<=$max_index; $i++) {
-                    $sql .= shift @values;
-                    $sql .= ',';
-                }
-                chop $sql;
-                print "$sql\n" if $dry;
-                print SQL "$sql;\n";
-            }
-        }
-#                       12     1 8      1    1= 23
+#my $queuedir = "/usr/home/queue";
-# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
+my $queuedir = "/usr/home/working";
-#                       12        1  8     1   1       3=26
+my $archivedir = "/usr/home/archive";
-# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
+my $sqldir = "/usr/home/sql";
-#                       12       1   8    1     1     5=28
+my $sqldirworking = "/usr/home/sql/tmp";
-# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
-        #print "finished at ";
-        #print `date`;
-        `mv $queuedir/$file $archivedir` unless $dry;
-        close(SQL);
-        `bzip2 $sqldirworking/$outfile`;
-        `mv $sqldirworking/$outfile.bz2 $sqldir/`;
-    }
-}
-`rm $queuedir/.lock` unless $dry;
-sub debug {
-    my $message = shift;
-    if ($debug) {
-        print "$message\n";
-    }
-}
-# var full during ft-v05.2005-03-28.084500-0800 and
-# 2005-02-24 69.55.226
-# all port/daily totals before 2005-04-07
-</pre>
-This script sends the sql files to the traffic server for processing:
-<pre>cat > /usr/home/flowbin/sendsql.pl
-#!/usr/bin/perl
-#use strict;
-#$debug=1;
-#$dry=1;
-my $remote = "69.55.233.199";
-my $sqldir = "/usr/home/sql";
-my $archive = "/usr/home/archive";
-my $sqldirremote = "/data/bwdb2/pending/";
-my @err;
 unless ($dry) {
-     if (-e "$sqldir/.lock") {
+     if (-e "$queuedir/.lock") {
-         open(FILE, "$sqldir/.lock");
+         open(FILE, "$queuedir/.lock");
          my $pid = <FILE>;
          chomp($pid);
@@ Line 2,195: / Line 2,302: @@
          else {
              #dead lock file, remove it
-             `rm $sqldir/.lock`;
+             `rm $queuedir/.lock`;
          }
      }
-     open(FILE, "> $sqldir/.lock");
+     open(FILE, "> $queuedir/.lock");
      print FILE "$$\n";
      close(FILE);
 }
-opendir(DIR, $sqldir);
+opendir(DIR, $queuedir);
 my @files = readdir(DIR);
 closedir(DIR);
 foreach my $file (sort @files) {
-   next unless $file =~ /bz2$/;
+    unless($file =~ /^\./) {
+        $file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
+        my $date = "$1 $2:$3:$4";
+        my $outfile = "$1-$2:$3.sql";
+        unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
+        my $condensedDate = $1;
+        $condensedDate =~ s/-//g;
+        my $iptotal = {};
+        my $protototal = {};
+        my $porttotal = {};
-   my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
+        &debug("started file $file at ");
-#   print "scp $sqldir/$file $remote:$sqldirremote";
+        &debug(`date`);
-   unless ($?==0) {
+        &debug("getting raw flow data (flow-print)");
-      push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
+        `cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
-   }
+        &debug("aggregating data at ");
-   else {
+        &debug(`date`);
-      `mv $sqldir/$file $archive`;
+        unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
-      `ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
+        LOOP: while (my $line = readline DATA) {
-   }
+            my @d = split /[\s]+/, $line;
-}
+            if ($d[0] ne '' && $d[0] ne 'Start') {
+                my $addr = 0;
+                my $port = 0;
-`rm $sqldir/.lock` unless $dry;
+                #Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
+                #0     1   2   3            4    5   6            7    8 9  10   11
+                          #|
+                          # outbound = 2, inbound = 1
-if (@err) {
+                my (@src_ip) = split '\.', $d[3];
-   email_support('bwdb2: sendsql.pl error',join "\n", @err);
+                my (@dst_ip) = split '\.', $d[6];
-}
-sub email_support {
+                if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
-    my $subj=shift;
+                   $d[2] = 2;
-    my $body=shift;
+                   # hack for outbound bulk traffic counted 2x
-    use Mail::Sendmail;
+                   #if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
+                }
+                # note- this is where we filter out IPs only found at i2b
+                elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
+                   $d[2] = 1;
+                }
+                else { next LOOP; }
-    # prepare message
+                if ($d[2] == 2) {
-    my %mail = (
+                    $addr = $d[3];
-        To      => 'support@johncompanies.com,dave@johncompanies.com',
+                    # if the dst-port is low, store that
-        From    => 'support@johncompanies.com',
+                    if ($d[7] <= 1024) { $port = $d[7]; }
-        Subject => $subj,
+                    # if the src-port is low, store that
-        Message => $body,
+                    elsif ($d[4] <= 1024) { $port = $d[4]; }
-        smtp    => 'mail.johncompanies.com',
+                    else { $port = 99999; }
-    );
+                }
-    sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
+                elsif ($d[2] == 1) {
-}
+                    $addr = $d[6];
+                    # if the dst-port is high, assume its return traffic, try to store src-port if low
-sub debug {
+                    if ($d[7] > 1024) {
-    my $message = shift;
+                        if ($d[4] <= 1024) { $port = $d[4]; }
-    if ($debug) {
+                        else { $port = 99999; }
-        print "$message\n";
+                    } else {
-    }
+                        $port = $d[7];
-}
+                    }
+                } else {
+                    next LOOP;
+                }
-# var full during ft-v05.2005-03-28.084500-0800 and
+                my (@ip) = split '\.', $addr;
-# 2005-02-24 69.55.226
+                unless ($ip[0] == 69) { next LOOP; }
+                unless ($ip[1] == 55) { next LOOP; }
+                unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }
-# all port/daily totals before 2005-04-07
+                my $classC = "$ip[0]_$ip[1]_$ip[2]";
-</pre>
-<pre>crontab -e
+#                          IP        dir
-#process flows
+#                if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
-,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
+#                if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
-#move sql commands to traffic db
+#
-,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
+#                if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
-</pre>
+#                if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
+#
+#                if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
+#                if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
+                $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
+                $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];
-==== flow processing: castle ====
+                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
-<pre>
+                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];
-cat > /usr/home/flowbin/processflows.pl
-#!/usr/bin/perl
+                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
+                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
-#use strict;
+            }
-#$debug=1;
+        }
-#$dry=1;
+        close(DATA);
+        `rm /usr/home/working/tmp-$file`;
+        &debug("processing ip totals at ");
+        &debug(`date`);
+        foreach my $classC (keys(%{$iptotal})) {
+            my @values;
+            foreach my $ip (keys(%{$iptotal->{$classC}})) {
+                foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
+                    my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
+                    my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
+#                    $packets = $packets > 2147483647 ? 0 : $packets;
+                    if ($octets > 2147483647) {
+                        my $ddir = $dir==1 ? 'in' : 'out';
+                        #print SQL "$date $ip $ddir $octets\n";
+#                        $octets = 0;
+                    }
+                    # dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
+                    my $id = "$ip-$condensedDate-$dir";
+                    $id =~ s/\.//g;
+                    push @values, "('$date', '$ip', $dir, $octets, $packets)";
+                    my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
+                    print "$sql\n" if $dry;
+                    print SQL "$sql;\n";
+#                    $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
+                }
+            }
-my $log = '/usr/home/flowbin/discards.log';
+            # break inserts into 100 records at a time
+            &debug("inserting $#values +1 values");
+            while ($#values > 0) {
+                my $sql = "insert into ipTotals_$classC values ";
+                my $max_index = $#values > 100 ? 100 : $#values;
+                for (my $i=0; $i<=$max_index; $i++) {
+                    $sql .= shift @values;
+                    $sql .= ',';
+                }
+                chop $sql;
+                print "$sql\n" if $dry;
+                print SQL "$sql;\n";
+            }
+        }
-use Data::Dumper;
+#        &debug("processing protocol totals at ");
+#        &debug(`date`);
+#        foreach my $classC (keys(%{$protototal})) {
+#            $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
+#            my @values;
+#            foreach my $ip (keys(%{$protototal->{$classC}})) {
+#                foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
+#                    foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
+#                        my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
+#                        my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
+# #                        $octets = $octets > 2147483647 ? 0 : $octets;
+# #                        $packets = $packets > 2147483647 ? 0 : $packets;
+#                        # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
+#                        my $id = "$ip-$condensedDate-$dir-$proto";
+#                        $id =~ s/\.//g;
+#                        push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
+#                        my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
+#                        print "$sql\n" if $dry;
+#                        $db->query($sql) unless $dry;
+# #                        $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
+#                    }
+#                }
+#            }
+#            $db->query("unlock tables") unless $dry;
+#            my $sql = "insert into protoTotals_$classC values ";
+#            $sql .= join ',', @values;
+#            $db->query("lock tables protoTotals_$classC write") unless $dry;
+#            print "$sql\n" if $dry;
+#            $db->query($sql) unless $dry;
+#            $db->query("unlock tables") unless $dry;
+#        }
-BEGIN {
+        &debug("processing port totals at ");
-    push @INC, "/usr/home/flowbin";
+        &debug(`date`);
-}
+        foreach my $classC (keys(%{$porttotal})) {
+            my @values;
-use db;
+            foreach my $ip (keys(%{$porttotal->{$classC}})) {
+                foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
-#my $queuedir = "/usr/home/queue";
+                    foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
-my $queuedir = "/usr/home/working";
+                        foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
-my $archivedir = "/usr/home/archive";
+                            my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
+                            my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
+    #                        $octets = $octets > 2147483647 ? 0 : $octets;
+    #                        $packets = $packets > 2147483647 ? 0 : $packets;
-unless ($dry) {
+                            # dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
-    if (-e "$queuedir/.lock") {
+                            my $id = "$ip-$condensedDate-$dir-$proto-$port";
-        open(FILE, "$queuedir/.lock");
+                            $id =~ s/\.//g;
-        my $pid = <FILE>;
+                            push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
-        chomp($pid);
+                            my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
-        close(FILE);
+                            print "$sql\n" if $dry;
-        if (kill(0, $pid)) {
+                            print SQL "$sql;\n";
-            #another process is using the queue, bail out
-            exit(0);
-        }
-        else {
-            #dead lock file, remove it
-            `rm $queuedir/.lock`;
-        }
-    }
-    open(FILE, "> $queuedir/.lock");
-    print FILE "$$\n";
-    close(FILE);
-}
-my $db = db->new();
+    #                        $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
-$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};
+                        }
+                    }
+                }
+            }
-opendir(DIR, $queuedir);
+            # break inserts into 100 records at a time
-my @files = readdir(DIR);
+            &debug("inserting $#values +1 values");
-closedir(DIR);
+            while ($#values > 0) {
+                my $sql = "insert into portTotals_$classC values ";
+                my $max_index = $#values > 100 ? 100 : $#values;
+                for (my $i=0; $i<=$max_index; $i++) {
+                    $sql .= shift @values;
+                    $sql .= ',';
+                }
+                chop $sql;
+                print "$sql\n" if $dry;
+                print SQL "$sql;\n";
+            }
+        }
-foreach my $file (sort @files) {
+#                       12     1 8      1    1= 23
-     unless($file =~ /^\./) {
+# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
-        $file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
+#                       12        1  8     1   1       3=26
-         my $date = "$1 $2:$3:$4";
+# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
-         my $condensedDate = $1;
+#                       12       1   8    1     1     5=28
-         $condensedDate =~ s/-//g;
+# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
-        my $iptotal = {};
+        #print "finished at ";
-        my $protototal = {};
+         #print `date`;
-        my $porttotal = {};
+        `mv $queuedir/$file $archivedir` unless $dry;
+        close(SQL);
+         `bzip2 $sqldirworking/$outfile`;
+         `mv $sqldirworking/$outfile.bz2 $sqldir/`;
+    }
+}
+`rm $queuedir/.lock` unless $dry;
-        &debug("started file $file at ");
+sub debug {
-        &debug(`date`);
+    my $message = shift;
-         &debug("getting raw flow data (flow-print)");
+    if ($debug) {
-        `cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
+         print "$message\n";
-        &debug("aggregating data at ");
+    }
-        &debug(`date`);
+}
-        unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
-        LOOP: while (my $line = readline DATA) {
+# var full during ft-v05.2005-03-28.084500-0800 and
-            my @d = split /[\s]+/, $line;
+# 2005-02-24 69.55.226
-            if ($d[0] ne '' && $d[0] ne 'Start') {
-                my $addr = 0;
-                my $port = 0;
-                #Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
+# all port/daily totals before 2005-04-07
-                #0     1   2   3            4    5   6            7    8 9  10   11
+</pre>
-                          #|
-                          # outbound = 2, inbound = 1
-                my (@src_ip) = split '\.', $d[3];
+This script sends the sql files to the traffic server for processing:
-                my (@dst_ip) = split '\.', $d[6];
+<pre>cat > /usr/home/flowbin/sendsql.pl
+#!/usr/bin/perl
-                if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
+#use strict;
-                    $src_ip[2] >= 224 && $src_ip[2] <= 239 &&
+#$debug=1;
-                    $src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
+#$dry=1;
-#                if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
-                   $d[2] = 2;
-                   # hack for outbound bulk traffic counted 2x
-                   if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
-                }
-                elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
-                       $dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
-                       $dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
-#                elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
-                   $d[2] = 1;
-                }
-                else { next LOOP; }
-                if ($d[2] == 2) {
+my $remote = "69.55.233.199";
-                    $addr = $d[3];
+my $sqldir = "/usr/home/sql";
-                    # if the dst-port is low, store that
+my $archive = "/usr/home/archive";
-                    if ($d[7] <= 1024) { $port = $d[7]; }
+my $sqldirremote = "/data/bwdb2/pending/";
-                    # if the src-port is low, store that
+my @err;
-                    elsif ($d[4] <= 1024) { $port = $d[4]; }
+unless ($dry) {
-                    else { $port = 99999; }
+    if (-e "$sqldir/.lock") {
-                }
+        open(FILE, "$sqldir/.lock");
-                elsif ($d[2] == 1) {
+        my $pid = <FILE>;
-                    $addr = $d[6];
+        chomp($pid);
-                    # if the dst-port is high, assume its return traffic, try to store src-port if low
+        close(FILE);
-                    if ($d[7] > 1024) {
+        if (kill(0, $pid)) {
-                        if ($d[4] <= 1024) { $port = $d[4]; }
+            #another process is using the queue, bail out
-                        else { $port = 99999; }
+            exit(0);
-                    } else {
+        }
-                        $port = $d[7];
+        else {
-                    }
+            #dead lock file, remove it
-                } else {
+            `rm $sqldir/.lock`;
-                    next LOOP;
+        }
-                }
+    }
+    open(FILE, "> $sqldir/.lock");
+    print FILE "$$\n";
+    close(FILE);
+}
-                my (@ip) = split '\.', $addr;
+opendir(DIR, $sqldir);
-                unless ($ip[0] == 69) { next LOOP; }
+my @files = readdir(DIR);
-                unless ($ip[1] == 55) { next LOOP; }
+closedir(DIR);
-                unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
-#                unless ($ip[2] == 229) { next LOOP; }
+foreach my $file (sort @files) {
+   next unless $file =~ /bz2$/;
-                my $classC = "$ip[0]_$ip[1]_$ip[2]";
+   my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
+#   print "scp $sqldir/$file $remote:$sqldirremote";
+   unless ($?==0) {
+      push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
+   }
+   else {
+      `mv $sqldir/$file $archive`;
+      `ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
+   }
+}
-#                          IP        dir
+`rm $sqldir/.lock` unless $dry;
-#                if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
-#                if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
-#
-#                if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
-#                if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
-#
-#                if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
-#                if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
-                $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
-                $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];
-                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
+if (@err) {
-                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];
+   email_support('bwdb2: sendsql.pl error',join "\n", @err);
+}
-                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
+sub email_support {
-                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
+    my $subj=shift;
-            }
+    my $body=shift;
-        }
+    use Mail::Sendmail;
-        close(DATA);
-        `rm /usr/home/working/tmp-$file`;
-        &debug("processing ip totals at ");
-        &debug(`date`);
-        foreach my $classC (keys(%{$iptotal})) {
-            $db->query("lock tables dailyIpTotals_$classC write") unless $dry;
-            my @values;
-            foreach my $ip (keys(%{$iptotal->{$classC}})) {
-                foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
-                    my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
-                    my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
-#                    $packets = $packets > 2147483647 ? 0 : $packets;
-                    if ($octets > 2147483647) {
-                        my $ddir = $dir==1 ? 'in' : 'out';
-                        `echo "$date $ip $ddir $octets\n" >> $log`;
-#                        $octets = 0;
-                    }
-                    # dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
-                    my $id = "$ip-$condensedDate-$dir";
-                    $id =~ s/\.//g;
-                    push @values, "('$date', '$ip', $dir, $octets, $packets)";
-                    my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
-                    print "$sql\n" if $dry;
-                    $db->query($sql) unless $dry;
-#                    $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
-                }
-            }
-            $db->query("unlock tables") unless $dry;
-            $db->query("lock tables ipTotals_$classC write") unless $dry;
+    # prepare message
-            # break inserts into 100 records at a time
+    my %mail = (
-            &debug("inserting $#values +1 values");
+        To      => 'support@johncompanies.com,dave@johncompanies.com',
-            while ($#values > 0) {
+        From    => 'support@johncompanies.com',
-                my $sql = "insert into ipTotals_$classC values ";
+        Subject => $subj,
-                my $max_index = $#values > 100 ? 100 : $#values;
+        Message => $body,
-                for (my $i=0; $i<=$max_index; $i++) {
+        smtp    => 'mail.johncompanies.com',
-                    $sql .= shift @values;
+    );
-                    $sql .= ',';
+    sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
-                }
+}
-                chop $sql;
-                print "$sql\n" if $dry;
-                $db->query($sql) unless $dry;
-            }
-            $db->query("unlock tables") unless $dry;
-        }
-        sleep 20;
+sub debug {
-#        &debug("processing protocol totals at ");
+    my $message = shift;
-#        &debug(`date`);
+    if ($debug) {
-#        foreach my $classC (keys(%{$protototal})) {
+        print "$message\n";
-#            $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
+    }
-#            my @values;
+}
-#            foreach my $ip (keys(%{$protototal->{$classC}})) {
-#                foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
-#                    foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
-#                        my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
-#                        my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
-# #                        $octets = $octets > 2147483647 ? 0 : $octets;
-# #                        $packets = $packets > 2147483647 ? 0 : $packets;
-#                        # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
-#                        my $id = "$ip-$condensedDate-$dir-$proto";
-#                        $id =~ s/\.//g;
-#                        push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
-#                        my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
-#                        print "$sql\n" if $dry;
-#                        $db->query($sql) unless $dry;
-# #                        $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
-#                    }
-#                }
-#            }
-#            $db->query("unlock tables") unless $dry;
-#            my $sql = "insert into protoTotals_$classC values ";
-#            $sql .= join ',', @values;
-#            $db->query("lock tables protoTotals_$classC write") unless $dry;
-#            print "$sql\n" if $dry;
-#            $db->query($sql) unless $dry;
-#            $db->query("unlock tables") unless $dry;
-#        }
-        &debug("processing port totals at ");
+# var full during ft-v05.2005-03-28.084500-0800 and
-        &debug(`date`);
+# 2005-02-24 69.55.226
-        foreach my $classC (keys(%{$porttotal})) {
-            $db->query("lock tables dailyPortTotals_$classC write") unless $dry;
+# all port/daily totals before 2005-04-07
-            my @values;
+</pre>
-            foreach my $ip (keys(%{$porttotal->{$classC}})) {
-                foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
-                    foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
-                        foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
-                            my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
-                            my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
-    #                        $octets = $octets > 2147483647 ? 0 : $octets;
-    #                        $packets = $packets > 2147483647 ? 0 : $packets;
-                            # dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
+<pre>crontab -e
-                            my $id = "$ip-$condensedDate-$dir-$proto-$port";
+#process flows
-                            $id =~ s/\.//g;
+,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
-                            push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
+#move sql commands to traffic db
-                            my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
+,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
-                            print "$sql\n" if $dry;
+</pre>
-                            $db->query($sql) unless $dry;
-    #                        $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
-                        }
-                    }
-                }
-            }
-            $db->query("unlock tables") unless $dry;
-            $db->query("lock tables portTotals_$classC write") unless $dry;
+==== flow processing: castle ====
-            # break inserts into 100 records at a time
+<pre>
-            &debug("inserting $#values +1 values");
+cat > /usr/home/flowbin/processflows.pl
-            while ($#values > 0) {
-                my $sql = "insert into portTotals_$classC values ";
-                my $max_index = $#values > 100 ? 100 : $#values;
-                for (my $i=0; $i<=$max_index; $i++) {
-                    $sql .= shift @values;
-                    $sql .= ',';
-                }
-                chop $sql;
-                print "$sql\n" if $dry;
-                $db->query($sql) unless $dry;
-            }
-            $db->query("unlock tables") unless $dry;
-            sleep 10;
-        }
-#                       12     1 8      1    1= 23
+#!/usr/bin/perl
-# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
-#                       12        1  8     1   1       3=26
-# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
-#                       12       1   8    1     1     5=28
-# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
-        #print "finished at ";
-        #print `date`;
-        `mv $queuedir/$file $archivedir` unless $dry;
-    }
-}
-`rm $queuedir/.lock` unless $dry;
-sub debug {
+#use strict;
-    my $message = shift;
+#$debug=1;
-    if ($debug) {
+#$dry=1;
-        print "$message\n";
-    }
-}
-# var full during ft-v05.2005-03-28.084500-0800 and
+my $log = '/usr/home/flowbin/discards.log';
-# 2005-02-24 69.55.226
-# all port/daily totals before 2005-04-07
+use Data::Dumper;
-</pre>
-<pre>
+BEGIN {
-cat > /usr/home/flowbin/db.pm
+    push @INC, "/usr/home/flowbin";
+}
-#!/usr/bin/perl
+use db;
-#
-# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
-#
-# Copyright (c) 2003
-#      e-Monitoring Networks, Inc.  All rights reserved.
-#
-#
-package db;
-use strict;
+#my $queuedir = "/usr/home/queue";
-use DBI;
+my $queuedir = "/usr/home/working";
+my $archivedir = "/usr/home/archive";
-sub new {
+unless ($dry) {
-     my $class = shift;
+     if (-e "$queuedir/.lock") {
-    my $self = {};
+        open(FILE, "$queuedir/.lock");
+        my $pid = <FILE>;
+        chomp($pid);
+        close(FILE);
+        if (kill(0, $pid)) {
+            #another process is using the queue, bail out
+            exit(0);
+        }
+        else {
+            #dead lock file, remove it
+            `rm $queuedir/.lock`;
+        }
+    }
+    open(FILE, "> $queuedir/.lock");
+    print FILE "$$\n";
+    close(FILE);
+}
-    $self->{'debug'} = 0;
+my $db = db->new();
-    bless $self, $class;
+$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};
-    return $self;
+opendir(DIR, $queuedir);
-}
+my @files = readdir(DIR);
+closedir(DIR);
-sub connect {
-    my $self = shift;
-    my $dbname = shift;
-    my $dbhost = shift;
-    my $dbuser = shift;
-    my $dbpass = shift;
-     my $host = '';
+foreach my $file (sort @files) {
-    if (defined($dbhost)) {
+     unless($file =~ /^\./) {
-         $host = ";host=$dbhost";
+        $file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
-    }
+         my $date = "$1 $2:$3:$4";
+        my $condensedDate = $1;
+        $condensedDate =~ s/-//g;
+        my $iptotal = {};
+        my $protototal = {};
+        my $porttotal = {};
-    eval {
+         &debug("started file $file at ");
-         $self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
+        &debug(`date`);
-         $self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
+        &debug("getting raw flow data (flow-print)");
-    };
+         `cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
-    if ($self->{'dbh'}) {
+        &debug("aggregating data at ");
-        return 1;
+        &debug(`date`);
-    }
+        unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
-    $self->{'error'} = "Error connecting to database $@";
+        LOOP: while (my $line = readline DATA) {
-    $self->debug("Error connecting to database $@");
+            my @d = split /[\s]+/, $line;
-    return 0;
+            if ($d[0] ne '' && $d[0] ne 'Start') {
-}
+                my $addr = 0;
+                my $port = 0;
-sub query {
+                #Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
-     my $self = shift;
+                #0     1   2   3            4    5   6            7    8 9  10   11
-    my $query = shift;
+                          #|
+                          # outbound = 2, inbound = 1
-    $self->debug($query, 1);
+                my (@src_ip) = split '\.', $d[3];
-    my $sth;
+                my (@dst_ip) = split '\.', $d[6];
-    eval {
-        $sth = $self->{'dbh'}->prepare($query);
-    };
-    unless ($sth) {
-        $self->{'error'} = "error preparing query $@";
-        $self->debug("error preparing query $@");
-        return undef;
-    }
-    my $qty;
-    eval {
-        $qty = $sth->execute;
-    };
-    unless ($qty) {
-        $self->{'error'} = "error executing query $@";
-        warn "error executing query $@ $query";
-        return undef;
-    }
-    $self->debug("returning $qty, $sth from query", 6);
-    return ($qty, $sth);
-}
-sub disconnect {
+                if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
-    my $self = shift;
+                    $src_ip[2] >= 224 && $src_ip[2] <= 239 &&
+                    $src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
-    $self->{'dbh'}->disconnect;
+#                if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
-    return 0;
+                   $d[2] = 2;
-}
+                   # hack for outbound bulk traffic counted 2x
+                   if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
-sub debug {
+                }
-    my $self = shift;
+                elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
-    my $msg = shift;
+                       $dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
-    my $level = shift || 0;
+                       $dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
+#                elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
+                   $d[2] = 1;
+                }
+                else { next LOOP; }
-    if ($level < $self->{'debug'}) {
+                if ($d[2] == 2) {
-        print "$msg\n";
+                    $addr = $d[3];
-    }
+                    # if the dst-port is low, store that
-    return 0;
+                    if ($d[7] <= 1024) { $port = $d[7]; }
-}
+                    # if the src-port is low, store that
-;
+                    elsif ($d[4] <= 1024) { $port = $d[4]; }
-</pre>
+                    else { $port = 99999; }
+                }
+                elsif ($d[2] == 1) {
+                    $addr = $d[6];
+                    # if the dst-port is high, assume its return traffic, try to store src-port if low
+                    if ($d[7] > 1024) {
+                        if ($d[4] <= 1024) { $port = $d[4]; }
+                        else { $port = 99999; }
+                    } else {
+                        $port = $d[7];
+                    }
+                } else {
+                    next LOOP;
+                }
- mkdir /usr/home/archive
+                my (@ip) = split '\.', $addr;
- mkdir -p /usr/home/sql/tmp
+                unless ($ip[0] == 69) { next LOOP; }
+                unless ($ip[1] == 55) { next LOOP; }
+                unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
+#                unless ($ip[2] == 229) { next LOOP; }
-<pre>crontab -e
+                my $classC = "$ip[0]_$ip[1]_$ip[2]";
-#process flows
-,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>
-==== setup traffic db ====
+#                          IP        dir
-* Install mysql:
+#                if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
-<pre>cd /usr/ports/databases/mysql50-server
+#                if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
-make install clean</pre>
+#
+#                if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
+#                if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
+#
+#                if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
+#                if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
+                $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
+                $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];
- cat >> /etc/rc.conf
+                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
- mysql_enable="YES"
+                $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];
-Move db data dir:
+                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
- /usr/local/etc/rc.d/mysql-server stop
+                $porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
- mkdir /usr/home/database/
+            }
- mv /var/db/mysql/* /usr/home/database/
+        }
- chown -R mysql:mysql /usr/home/database
+        close(DATA);
+        `rm /usr/home/working/tmp-$file`;
-Edit database location in startup script:
+        &debug("processing ip totals at ");
- vi /usr/local/etc/rc.d/mysql-server
+        &debug(`date`);
- # : ${mysql_dbdir="/var/db/mysql"}
+        foreach my $classC (keys(%{$iptotal})) {
- : ${mysql_dbdir="/usr/home/database"}
+            $db->query("lock tables dailyIpTotals_$classC write") unless $dry;
+            my @values;
- /usr/local/etc/rc.d/mysql-server start
+            foreach my $ip (keys(%{$iptotal->{$classC}})) {
+                foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
+                    my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
+                    my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
+#                    $packets = $packets > 2147483647 ? 0 : $packets;
+                    if ($octets > 2147483647) {
+                        my $ddir = $dir==1 ? 'in' : 'out';
+                        `echo "$date $ip $ddir $octets\n" >> $log`;
+#                        $octets = 0;
+                    }
+                    # dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
+                    my $id = "$ip-$condensedDate-$dir";
+                    $id =~ s/\.//g;
+                    push @values, "('$date', '$ip', $dir, $octets, $packets)";
+                    my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
+                    print "$sql\n" if $dry;
+                    $db->query($sql) unless $dry;
+#                    $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
+                }
+            }
+            $db->query("unlock tables") unless $dry;
+            $db->query("lock tables ipTotals_$classC write") unless $dry;
+            # break inserts into 100 records at a time
+            &debug("inserting $#values +1 values");
+            while ($#values > 0) {
+                my $sql = "insert into ipTotals_$classC values ";
+                my $max_index = $#values > 100 ? 100 : $#values;
+                for (my $i=0; $i<=$max_index; $i++) {
+                    $sql .= shift @values;
+                    $sql .= ',';
+                }
+                chop $sql;
+                print "$sql\n" if $dry;
+                $db->query($sql) unless $dry;
+            }
+            $db->query("unlock tables") unless $dry;
+        }
-* Install mysql perl database modules:
+        sleep 20;
-<pre>
+#        &debug("processing protocol totals at ");
-cd /usr/ports/databases/p5-DBI
+#        &debug(`date`);
-make install clean
+#        foreach my $classC (keys(%{$protototal})) {
-cd /usr/ports/databases/p5-DBD-mysql50
+#            $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
-make install clean
+#            my @values;
-(no to SSL support)
+#            foreach my $ip (keys(%{$protototal->{$classC}})) {
-</pre>
+#                foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
+#                    foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
+#                        my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
+#                        my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
+# #                        $octets = $octets > 2147483647 ? 0 : $octets;
+# #                        $packets = $packets > 2147483647 ? 0 : $packets;
+#                        # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
+#                        my $id = "$ip-$condensedDate-$dir-$proto";
+#                        $id =~ s/\.//g;
+#                        push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
+#                        my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
+#                        print "$sql\n" if $dry;
+#                        $db->query($sql) unless $dry;
+# #                        $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
+#                    }
+#                }
+#            }
+#            $db->query("unlock tables") unless $dry;
+#            my $sql = "insert into protoTotals_$classC values ";
+#            $sql .= join ',', @values;
+#            $db->query("lock tables protoTotals_$classC write") unless $dry;
+#            print "$sql\n" if $dry;
+#            $db->query($sql) unless $dry;
+#            $db->query("unlock tables") unless $dry;
+#        }
-* Setting up database
+        &debug("processing port totals at ");
-<pre>
+        &debug(`date`);
-rehash
+        foreach my $classC (keys(%{$porttotal})) {
-/usr/local/etc/rc.d/mysql-server start
+            $db->query("lock tables dailyPortTotals_$classC write") unless $dry;
-mysql -u root
+            my @values;
-create database traffic;
+            foreach my $ip (keys(%{$porttotal->{$classC}})) {
-grant all on *.* to root@localhost identified by '5over3';
+                foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
-grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';
+                    foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
+                        foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
+                            my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
+                            my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
+    #                        $octets = $octets > 2147483647 ? 0 : $octets;
+    #                        $packets = $packets > 2147483647 ? 0 : $packets;
-</pre>
+                            # dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
+                            my $id = "$ip-$condensedDate-$dir-$proto-$port";
+                            $id =~ s/\.//g;
+                            push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
+                            my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
+                            print "$sql\n" if $dry;
+                            $db->query($sql) unless $dry;
+    #                        $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
+                        }
+                    }
+                }
+            }
+            $db->query("unlock tables") unless $dry;
-If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.
+            $db->query("lock tables portTotals_$classC write") unless $dry;
+            # break inserts into 100 records at a time
+            &debug("inserting $#values +1 values");
+            while ($#values > 0) {
+                my $sql = "insert into portTotals_$classC values ";
+                my $max_index = $#values > 100 ? 100 : $#values;
+                for (my $i=0; $i<=$max_index; $i++) {
+                    $sql .= shift @values;
+                    $sql .= ',';
+                }
+                chop $sql;
+                print "$sql\n" if $dry;
+                $db->query($sql) unless $dry;
+            }
+            $db->query("unlock tables") unless $dry;
+            sleep 10;
+        }
-We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
+#                       12     1 8      1    1= 23
- rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/
+# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
+#                       12        1  8     1   1       3=26
+# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
+#                       12       1   8    1     1     5=28
+# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
+        #print "finished at ";
+        #print `date`;
+        `mv $queuedir/$file $archivedir` unless $dry;
+    }
+}
+`rm $queuedir/.lock` unless $dry;
-When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.
+sub debug {
+    my $message = shift;
+    if ($debug) {
+        print "$message\n";
+    }
+}
+# var full during ft-v05.2005-03-28.084500-0800 and
+# 2005-02-24 69.55.226
+# all port/daily totals before 2005-04-07
+</pre>
-==== process flows from bwdb2 ====
+<pre>
-On traffic database server (bwdb):
+cat > /usr/home/flowbin/db.pm
-<pre>crontab -e
+#!/usr/bin/perl
-#import sql from bwdb2
+#
-,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>
+# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
+#
+# Copyright (c) 2003
+#      e-Monitoring Networks, Inc.  All rights reserved.
+#
+#
+package db;
-Add access to mysql:
+use strict;
-<pre>mysql -u root -p
+use DBI;
-grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
-</pre>
-<pre>cat > /usr/home/flowbin/processsql.pl
+sub new {
+    my $class = shift;
-#!/usr/bin/perl
+    my $self = {};
-#use strict;
+    $self->{'debug'} = 0;
-#$debug=1;
+    bless $self, $class;
-#$dry=1;
-my $sqldir = "/usr/home/bwdb2/pending";
+     return $self;
-my $mysql = '/usr/local/bin/mysql';
-my @err;
-unless ($dry) {
-     if (-e "$sqldir/.lock") {
-        open(FILE, "$sqldir/.lock");
-        my $pid = <FILE>;
-        chomp($pid);
-        close(FILE);
-        if (kill(0, $pid)) {
-            #another process is using the queue, bail out
-            exit(0);
-        }
-        else {
-            #dead lock file, remove it
-            `rm $sqldir/.lock`;
-        }
-    }
-    open(FILE, "> $sqldir/.lock");
-    print FILE "$$\n";
-    close(FILE);
 }
-opendir(DIR, $sqldir);
+sub connect {
-my @files = readdir(DIR);
+    my $self = shift;
-closedir(DIR);
+    my $dbname = shift;
+    my $dbhost = shift;
+    my $dbuser = shift;
+    my $dbpass = shift;
-foreach my $file (sort @files) {
+    my $host = '';
-   next unless $file =~ /done$/;
+    if (defined($dbhost)) {
-   my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
+        $host = ";host=$dbhost";
-   unless ($?==0) {
+    }
-      push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
-   }
-   else {
-      `rm $sqldir/$file`;
-   }
-}
-`rm $sqldir/.lock` unless $dry;
+    eval {
+        $self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
-if (@err) {
+        $self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
-   email_support('bwdb: processsql.pl error',join "\n", @err);
+    };
+    if ($self->{'dbh'}) {
+        return 1;
+    }
+    $self->{'error'} = "Error connecting to database $@";
+    $self->debug("Error connecting to database $@");
+    return 0;
 }
-sub email_support {
+sub query {
-     my $subj=shift;
+     my $self = shift;
-     my $body=shift;
+     my $query = shift;
-    use Mail::Sendmail;
-     # prepare message
+     $self->debug($query, 1);
-     my %mail = (
+     my $sth;
-         To      => 'dave@johncompanies.com',
+    eval {
-         From    => 'support@johncompanies.com',
+         $sth = $self->{'dbh'}->prepare($query);
-         Subject => $subj,
+    };
-         Message => $body,
+    unless ($sth) {
-         smtp    => 'mail.johncompanies.com',
+         $self->{'error'} = "error preparing query $@";
-     );
+         $self->debug("error preparing query $@");
-     sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
+        return undef;
+    }
+    my $qty;
+    eval {
+         $qty = $sth->execute;
+    };
+    unless ($qty) {
+         $self->{'error'} = "error executing query $@";
+        warn "error executing query $@ $query";
+        return undef;
+    }
+     $self->debug("returning $qty, $sth from query", 6);
+     return ($qty, $sth);
+}
+sub disconnect {
+    my $self = shift;
+    $self->{'dbh'}->disconnect;
+    return 0;
 }
 sub debug {
-     my $message = shift;
+     my $self = shift;
-     if ($debug) {
+    my $msg = shift;
-         print "$message\n";
+    my $level = shift || 0;
+     if ($level < $self->{'debug'}) {
+         print "$msg\n";
      }
+    return 0;
 }
+;
 </pre>
-  chmod 0700 /usr/home/flowbin/processsql.pl
+  mkdir /usr/home/archive
+ mkdir -p /usr/home/sql/tmp
-Make sure bwdb is reachable from the outside only to bwdb2:
+<pre>crontab -e
+#process flows
+,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>
-On nat, add to <tt>/etc/ipnat.rules</tt>
+==== setup traffic db ====
-<pre># bwdb
+* Install mysql:
-bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>
+<pre>cd /usr/ports/databases/mysql50-server
+make install clean</pre>
-Reload:
+  cat >> /etc/rc.conf
-  ipnat -C -F -f /etc/ipnat.rules
+ mysql_enable="YES"
-Setup firewall rule on firewall:
+Move db data dir:
-  ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
+  /usr/local/etc/rc.d/mysql-server stop
-  ipfw add 00094 deny ip from any to 69.55.233.199
+ mkdir /usr/home/database/
+ mv /var/db/mysql/* /usr/home/database/
+  chown -R mysql:mysql /usr/home/database
-Setup firewall on bwdb to restrict access now that it's nat'd:
+Edit database location in startup script:
-<pre>
+ vi /usr/local/etc/rc.d/mysql-server
-cat >> /usr/local/etc/rc.d/boot.sh
+ # : ${mysql_dbdir="/var/db/mysql"}
-ipfw add 1 allow tcp from any to any established
+ : ${mysql_dbdir="/usr/home/database"}
-ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
-ipfw add 3 allow ip from 10.1.4.5 to me 3306
-ipfw add 4 allow ip from 69.55.225.225 53 to me
-ipfw add 5 allow ip from 69.55.230.2 25 to me
-ipfw add 6 allow ip from me to me 4444
-ipfw add 7 allow icmp from any to me
-ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
-ipfw add 9 allow udp from 10.1.4.5 to me 161
-ipfw add 100 deny ip from any to me
-</pre>
-  chmod 0700 /usr/local/etc/rc.d/boot.sh
+  /usr/local/etc/rc.d/mysql-server start
-From bwdb2, add ssh key:
- cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'
-Confirm no password access:
+* Install mysql perl database modules:
- ssh 69.55.233.199 hostname
+<pre>
+cd /usr/ports/databases/p5-DBI
-= bwdb2 =
+make install clean
-== Summary ==
+cd /usr/ports/databases/p5-DBD-mysql50
-This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.
+make install clean
+(no to SSL support)
+</pre>
-* Location: i2b, cab6
+* Setting up database
-* OS: FreeBSD 6.4 x86
+<pre>
-* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
+rehash
-* Hardware: Custom 2U. Single power supply.
+/usr/local/etc/rc.d/mysql-server start
-* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.
+mysql -u root
+create database traffic;
+grant all on *.* to root@localhost identified by '5over3';
+grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';
-== Services Provided ==
+</pre>
-* netflow
-* bigbrother
-== netflow ==
+If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.
-The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.
+We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
+ rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/
-A cronjob moves that flow file (or files if there are multiple due to some delay)
+When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.
-,16,31,46 * * * * /usr/home/flowbin/queue.pl
-into a processing queue:
-<tt>/usr/home/working</tt>
-Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
-,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
-Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
+==== process flows from bwdb2 ====
-,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
+On traffic database server (bwdb):
-== Regular maintenance ==
+<pre>crontab -e
-*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
+#import sql from bwdb2
+,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>
-* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>
+Add access to mysql:
+<pre>mysql -u root -p
+grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
+</pre>
-= firewall (newgateway) =
+<pre>cat > /usr/home/flowbin/processsql.pl
-== Summary ==
+#!/usr/bin/perl
-This machine is the primary (only) firewall for the entire network at castle.
+#use strict;
+#$debug=1;
+#$dry=1;
-* Location: castle, cab 3-8
+my $sqldir = "/usr/home/bwdb2/pending";
-* OS: FreeBSD 4.11 x86
+my $mysql = '/usr/local/bin/mysql';
-* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
+my @err;
-* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+unless ($dry) {
-* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.
+    if (-e "$sqldir/.lock") {
+        open(FILE, "$sqldir/.lock");
+        my $pid = <FILE>;
+        chomp($pid);
+        close(FILE);
+        if (kill(0, $pid)) {
+            #another process is using the queue, bail out
+            exit(0);
+        }
+        else {
+            #dead lock file, remove it
+            `rm $sqldir/.lock`;
+        }
+    }
+    open(FILE, "> $sqldir/.lock");
+    print FILE "$$\n";
+    close(FILE);
+}
-== Services Provided ==
+opendir(DIR, $sqldir);
-* firewall (ipfw)
+my @files = readdir(DIR);
-* snmp
+closedir(DIR);
-* bigbrother
-== Firewall Rule Configuration ==
+foreach my $file (sort @files) {
+   next unless $file =~ /done$/;
+   my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
+   unless ($?==0) {
+      push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
+   }
+   else {
+      `rm $sqldir/$file`;
+   }
+}
-See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.
+`rm $sqldir/.lock` unless $dry;
-== Disaster Recovery ==
+if (@err) {
+   email_support('bwdb: processsql.pl error',join "\n", @err);
+}
-If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.
+sub email_support {
+    my $subj=shift;
+    my $body=shift;
+    use Mail::Sendmail;
-Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)
+    # prepare message
+    my %mail = (
+        To      => 'dave@johncompanies.com',
+        From    => 'support@johncompanies.com',
+        Subject => $subj,
+        Message => $body,
+        smtp    => 'mail.johncompanies.com',
+    );
+    sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
+}
-<pre>hostname="newgateway.johncompanies.com"
+sub debug {
-firewall_script="/etc/firewall.sh"
+    my $message = shift;
-firewall_enable="NO"
+    if ($debug) {
-sendmail_enable="NONE"
+        print "$message\n";
-sshd_enable="YES"
+    }
-inetd_enable="NO"
+}
-xntpd_enable="YES"
+</pre>
-snmpd_enable="YES"
-#snmpd_flags="-as -p /var/run/snmpd.pid"
-#ipnat_enable="YES"
-#ipnat_rules="/etc/ipnat.rules"
-gateway_enable="YES"
-defaultrouter="69.55.233.161"
+ chmod 0700 /usr/home/flowbin/processsql.pl
-ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
+Make sure bwdb is reachable from the outside only to bwdb2:
-ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"
-#
+On nat, add to <tt>/etc/ipnat.rules</tt>
-# Original JohnCompanies 69.55.224.0/20
+<pre># bwdb
-#
+bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>
-ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"
-static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
+Reload:
-route13 route14 route15 route16 route17 route18"
+ ipnat -C -F -f /etc/ipnat.rules
-route_route1="-net 69.55.224.0 69.55.233.153"
+Setup firewall rule on firewall:
-route_route2="-net 69.55.225.0 69.55.233.153"
+ ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
-route_route3="-net 69.55.226.0 69.55.233.153"
+ ipfw add 00094 deny ip from any to 69.55.233.199
-route_route4="-net 69.55.227.0 69.55.233.153"
-route_route5="-net 69.55.228.0 69.55.233.153"
-route_route6="-net 69.55.229.0 69.55.233.153"
-route_route7="-net 69.55.230.0 69.55.233.153"
-route_route8="-net 69.55.231.0 69.55.233.153"
-route_route9="-net 69.55.232.0 69.55.233.153"
-route_route10="-net 69.55.233.0 69.55.233.153"
-route_route11="-net 69.55.234.0 69.55.233.153"
-route_route12="-net 69.55.235.0 69.55.233.153"
-route_route13="-net 69.55.236.0 69.55.233.153"
-route_route14="-net 69.55.237.0 69.55.233.153"
-route_route15="-net 69.55.238.0 69.55.233.153"
-route_route16="-net 69.55.239.0 69.55.233.153"
-route_route17="-net 10.1.5.0 10.1.4.2"
-route_route18="-net 10.1.6.0 10.1.4.2"
+Setup firewall on bwdb to restrict access now that it's nat'd:
+<pre>
+cat >> /usr/local/etc/rc.d/boot.sh
+ipfw add 1 allow tcp from any to any established
+ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
+ipfw add 3 allow ip from 10.1.4.5 to me 3306
+ipfw add 4 allow ip from 69.55.225.225 53 to me
+ipfw add 5 allow ip from 69.55.230.2 25 to me
+ipfw add 6 allow ip from me to me 4444
+ipfw add 7 allow icmp from any to me
+ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
+ipfw add 9 allow udp from 10.1.4.5 to me 161
+ipfw add 100 deny ip from any to me
+</pre>
-#In case of 3750 failure:
+ chmod 0700 /usr/local/etc/rc.d/boot.sh
-#defaultrouter="69.43.128.81"
-#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"
-#bind .1's here:
+From bwdb2, add ssh key:
-#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
+ cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'
-#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
-#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
-#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
-#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
-#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
-#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
-#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
-#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
-#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
-#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
-#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
-#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
-#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
-#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
-#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"
-#bulk:
+Confirm no password access:
-# reassign 69.55.231.1 to the int iface on the firewall
+ ssh 69.55.233.199 hostname
-# set the DG on the firewall to 69.43.138.9
-# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>
-== Cronjobs ==
+= bwdb2 =
-0 * * * /usr/local/etc/rsync.backup
+== Summary ==
-Backup to backup1
+This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.
-0 1 * * /sbin/ipfw zero
+* Location: i2b, cab6
-0 1 * * /sbin/ipfw del 3  4 5 17331
+* OS: FreeBSD 6.4 x86
-Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).
+* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
+* Hardware: Custom 2U. Single power supply.
+* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.
-Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
+== Services Provided ==
-This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.
+* netflow
+* bigbrother
-== DOS attacks ==
+== netflow ==
-See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.
+The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.
-Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
+A cronjob moves that flow file (or files if there are multiple due to some delay)
-# enter a rule that allows all UDP to go through
+,16,31,46 * * * * /usr/home/flowbin/queue.pl
-# send an emergency email to support and indicating an attack is in progress
-# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
-# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
-This file lives under /usr/home/user/doswatch.pl
-To run:
-  cd /usr/home/user
- ./doswatch.pl &
-To kill;
+into a processing queue:
- fg
+<tt>/usr/home/working</tt>
- ^C
-It writes its findings to /usr/home/user/doswatch.log
+Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
+,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
-= backup1 =
+Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
+,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
-== Summary ==
+== Regular maintenance ==
+*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
-This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.
+* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>
+= firewall (newgateway) =
+== Summary ==
+This machine is the primary (only) firewall for the entire network at castle.
 * Location: castle, cab 3-8
-* OS: Ubuntu 8.04.1 server x86
+* OS: FreeBSD 4.11 x86
-* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
+* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
-* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
+* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
-* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card
+* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.
-== Services provided ==
+== Services Provided ==
-* backup via rsync
+* firewall (ipfw)
-* mysql
-* nfs
 * snmp
 * bigbrother
-== Usage and Notes ==
+== Firewall Rule Configuration ==
-* all data is stored under /data
-* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
-* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
-* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
-* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
-* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
-* critical files from backup2 are stored under <tt>/data/backup2</tt>
-== Cronjobs ==
+See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.
-<pre>
-5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
-5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate  /data/backuplog.log</pre>
-this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.
-<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
+== Disaster Recovery ==
-</pre>
-this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
-<pre>0 0 * * * /usr/local/sbin/3wraidchk
-</pre>
-checks the health of the RAID array
-== Regular maintenance ==
+If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.
-*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
-*[[Routine_Maintenance#3ware|Check on auto-verify]]
-== build ==
+Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)
-<pre>Setup raid5 with a boot vol of 12G 5.45tb
+<pre>hostname="newgateway.johncompanies.com"
-G boot
+firewall_script="/etc/firewall.sh"
-GB
+firewall_enable="NO"
+sendmail_enable="NONE"
+sshd_enable="YES"
+inetd_enable="NO"
+xntpd_enable="YES"
+snmpd_enable="YES"
+#snmpd_flags="-as -p /var/run/snmpd.pid"
+#ipnat_enable="YES"
+#ipnat_rules="/etc/ipnat.rules"
+gateway_enable="YES"
-Install ubuntu 8.04
+defaultrouter="69.55.233.161"
-Swap 4G
+ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
+ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"
-Don’t format data drive
+#
+# Original JohnCompanies 69.55.224.0/20
+#
+ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"
-http://www.unixgods.org/~tilo/linux_larger_2TB.html
+static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
+route13 route14 route15 route16 route17 route18"
-parted /dev/sdb
+route_route1="-net 69.55.224.0 69.55.233.153"
-print
+route_route2="-net 69.55.225.0 69.55.233.153"
-mklabel gpt
+route_route3="-net 69.55.226.0 69.55.233.153"
-print
+route_route4="-net 69.55.227.0 69.55.233.153"
+route_route5="-net 69.55.228.0 69.55.233.153"
-#Disk /dev/sdb: 4987GB
+route_route6="-net 69.55.229.0 69.55.233.153"
-#Sector size (logical/physical): 512B/512B
+route_route7="-net 69.55.230.0 69.55.233.153"
-#Partition Table: gpt
+route_route8="-net 69.55.231.0 69.55.233.153"
+route_route9="-net 69.55.232.0 69.55.233.153"
+route_route10="-net 69.55.233.0 69.55.233.153"
+route_route11="-net 69.55.234.0 69.55.233.153"
+route_route12="-net 69.55.235.0 69.55.233.153"
+route_route13="-net 69.55.236.0 69.55.233.153"
+route_route14="-net 69.55.237.0 69.55.233.153"
+route_route15="-net 69.55.238.0 69.55.233.153"
+route_route16="-net 69.55.239.0 69.55.233.153"
+route_route17="-net 10.1.5.0 10.1.4.2"
+route_route18="-net 10.1.6.0 10.1.4.2"
-#Number  Start  End  Size  File system  Name  Flags
-mkpart primary ext3 0 4987GB
+#In case of 3750 failure:
-print
+#defaultrouter="69.43.128.81"
+#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"
-#Disk /dev/sdb: 5987GB
+#bind .1's here:
-#Sector size (logical/physical): 512B/512B
+#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
-#Partition Table: gpt
+#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
+#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
+#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
+#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
+#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
+#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
+#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
+#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
+#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
+#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
+#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
+#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
+#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
+#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
+#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"
-#Number  Start   End     Size    File system  Name     Flags
+#bulk:
-# 1      17.4kB  4987GB  4987GB               primary
+# reassign 69.55.231.1 to the int iface on the firewall
+# set the DG on the firewall to 69.43.138.9
+# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>
-quit
+== Cronjobs ==
+0 * * * /usr/local/etc/rsync.backup
+Backup to backup1
-mkfs.ext3 /dev/sdb1
+0 1 * * /sbin/ipfw zero
-#mke2fs 1.40.8 (13-Mar-2008)
+0 1 * * /sbin/ipfw del 3  4 5 17331
-#Filesystem label=
+Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).
-#OS type: Linux
-#Block size=4096 (log=2)
-#Fragment size=4096 (log=2)
-#304390144 inodes, 1217544183 blocks
-#60877209 blocks (5.00%) reserved for the super user
-#First data block=0
-#Maximum filesystem blocks=0
-#37157 block groups
-#32768 blocks per group, 32768 fragments per group
-#8192 inodes per group
-#Superblock backups stored on blocks:
-#        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
-#        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
-#        102400000, 214990848, 512000000, 550731776, 644972544
-#
-#Writing inode tables:   967/37157
-mkdir /data
+Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
+This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.
-#root@backup1:~# df -h
+== DOS attacks ==
-#Filesystem            Size  Used Avail Use% Mounted on
-#/dev/sda2             8.3G  540M  7.3G   7% /
-#varrun               1013M   40K 1013M   1% /var/run
-#varlock              1013M     0 1013M   0% /var/lock
-#udev                 1013M   56K 1013M   1% /dev
-#devshm               1013M     0 1013M   0% /dev/shm
-#/dev/sdb1             4.5T  192M  4.3T   1% /data
+See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.
-apt-get update
+Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
-apt-get upgrade
+# enter a rule that allows all UDP to go through
-apt-get install snmp snmpd ntp nfs-kernel-server
+# send an emergency email to support and indicating an attack is in progress
+# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
+# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
+This file lives under /usr/home/user/doswatch.pl
+To run:
+ cd /usr/home/user
+ ./doswatch.pl &
-echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
+To kill;
-echo "\"\e[6~\": history-search-forward" >> ~/.inputrc
+ fg
+ ^C
-vi /etc/ntp.conf
+It writes its findings to /usr/home/user/doswatch.log
-server 10.1.4.5
-scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
+= backup1 =
-cd /root/
-ssh-keygen -t dsa
-echo "10.1.4.3        backup2" >> /etc/hosts
-cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
+== Summary ==
-ssh backup2
+This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.
-vi /root/.bashrc
+* Location: castle, cab 3-8
-export PS1="[\u@\h \w]# "
+* OS: Ubuntu 8.04.1 server x86
-alias h='history'
+* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
-alias vi='vim'
+* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
-alias j='jobs'
+* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card
-export PS1="[\u@\h \w]# "
-alias dr='screen -dr'
-export EDITOR=vim
-export GREP_OPTIONS='--color=auto'
-export HISTFILESIZE=1000
-source /root/.bashrc
+== Services provided ==
+* backup via rsync
+* mysql - traffic data
+* nfs server - for backups
+* snmp client - for big brother
+* bigbrother client
-echo "# ttyS0 - getty
+== Usage and Notes ==
-#
+* all data is stored under /data
-# This service maintains a getty on ttyS0 from the point the system is
+* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
-# started until it is shut down again.
+* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
+* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
+* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
+* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
+* critical files from backup2 are stored under <tt>/data/backup2</tt>
-start on runlevel 2
+== Cronjobs ==
-start on runlevel 3
+<pre>
-start on runlevel 4
+5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
-start on runlevel 5
+5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate  /data/backuplog.log</pre>
+this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.
-stop on runlevel 0
+<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
-stop on runlevel 1
+</pre>
-stop on runlevel 6
+this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
+<pre>0 0 * * * /usr/local/sbin/3wraidchk
+</pre>
+checks the health of the RAID array
-respawn
+== Regular maintenance ==
-exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0
+*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
+*[[Routine_Maintenance#3ware|Check on auto-verify]]
+== build ==
-vi /boot/grub/menu.lst
+<pre>Setup raid5 with a boot vol of 12G 5.45tb
+G boot
+GB
-serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
+Install ubuntu 8.04
-terminal --timeout=15 serial console
-append to kernel lines:
+Swap 4G
-console=tty0 console=ttyS0,38400n8
-show menu:
+Don’t format data drive
-#hiddenmenu
-echo 'rocommunity  jcread 10.1.4.5
+http://www.unixgods.org/~tilo/linux_larger_2TB.html
-rocommunity  jcread 10.1.4.3
-agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf
-# to see which iface it is, on backup2:
+parted /dev/sdb
+print
+mklabel gpt
+print
-snmpwalk -v 1 -c jcread 10.1.4.8 interface
+#Disk /dev/sdb: 4987GB
+#Sector size (logical/physical): 512B/512B
+#Partition Table: gpt
+#Number  Start  End  Size  File system  Name  Flags
-echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
+mkpart primary ext3 0 4987GB
+print
-echo "bb:x:1984:" >> /etc/group
+#Disk /dev/sdb: 5987GB
+#Sector size (logical/physical): 512B/512B
+#Partition Table: gpt
-pwconv
+#Number  Start   End     Size    File system  Name     Flags
+# 1      17.4kB  4987GB  4987GB               primary
-mkdir /home/bb
+quit
-chown bb.bb /home/bb
-cd ~bb
+mkfs.ext3 /dev/sdb1
-scp backup2:/mnt/data4/build/bb/bb-linux.tar .
+#mke2fs 1.40.8 (13-Mar-2008)
+#Filesystem label=
-tar xf bb-linux.tar
+#OS type: Linux
+#Block size=4096 (log=2)
-cd /home/bb/bbc1.9e-btf/etc
+#Fragment size=4096 (log=2)
+#304390144 inodes, 1217544183 blocks
-echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
+#60877209 blocks (5.00%) reserved for the super user
-echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts
+#First data block=0
+#Maximum filesystem blocks=0
+#37157 block groups
+#32768 blocks per group, 32768 fragments per group
+#8192 inodes per group
+#Superblock backups stored on blocks:
+#        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
+#        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
+#        102400000, 214990848, 512000000, 550731776, 644972544
+#
+#Writing inode tables:   967/37157
+mkdir /data
-echo "/:90:95
+#root@backup1:~# df -h
-/var:90:95
+#Filesystem            Size  Used Avail Use% Mounted on
-/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab
+#/dev/sda2             8.3G  540M  7.3G   7% /
+#varrun               1013M   40K 1013M   1% /var/run
+#varlock              1013M     0 1013M   0% /var/lock
+#udev                 1013M   56K 1013M   1% /dev
+#devshm               1013M     0 1013M   0% /dev/shm
+#/dev/sdb1             4.5T  192M  4.3T   1% /data
-vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
+apt-get update
-(remove all | SORT xxxx)
+apt-get upgrade
+apt-get install snmp snmpd ntp nfs-kernel-server
-chmod +r /var/log/messages
+echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
+echo "\"\e[6~\": history-search-forward" >> ~/.inputrc
-./bbchkcfg.sh
+vi /etc/ntp.conf
-#(y to questions)
+server 10.1.4.5
-./bbchkhosts.sh
-#(ignore ssh errors)
-cd ../..
-chown -R bb .
-su bb
-cd
-cd bbc1.9e-btf/src
-#make; make install
+scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
-cd ..
+cd /root/
-./runbb.sh start
+ssh-keygen -t dsa
-more BBOUT
+echo "10.1.4.3        backup2" >> /etc/hosts
-(look for errors)
-exit
-vi /etc/rc.local
+cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
-su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
+ssh backup2
-echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports
+vi /root/.bashrc
+export PS1="[\u@\h \w]# "
+alias h='history'
+alias vi='vim'
+alias j='jobs'
+export PS1="[\u@\h \w]# "
+alias dr='screen -dr'
+export EDITOR=vim
+export GREP_OPTIONS='--color=auto'
+export HISTFILESIZE=1000
-/etc/init.d/nfs-kernel-server restart
+source /root/.bashrc
+echo "# ttyS0 - getty
+#
+# This service maintains a getty on ttyS0 from the point the system is
+# started until it is shut down again.
-echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd
+start on runlevel 2
+start on runlevel 3
+start on runlevel 4
+start on runlevel 5
+stop on runlevel 0
+stop on runlevel 1
+stop on runlevel 6
-echo '10.1.4.8                backup1' >> /etc/hosts
+respawn
-echo '/dev/sdb1	/data  ext3  rw,noatime  0  0' >> /etc/fstab
+exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0
-to install digi drivers:
+vi /boot/grub/menu.lst
-wget http://ftp1.digi.com/support/driver/40002086_n.tgz
+serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
-apt-get install linux-image-2.6.24-19-server
+terminal --timeout=15 serial console
-apt-get install linux-source-2.6.24 (not needed?)
-apt-get install linux-headers-2.6.24-19-server
-apt-get install make
-apt-get install gcc
-apt-get install g++
-apt-get install libncurses5-dev
-apt-get install expect
-apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl
-cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
+append to kernel lines:
-./configure
+console=tty0 console=ttyS0,38400n8
-make all
-make install
-make postinstall
-/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8
+show menu:
+#hiddenmenu
-apt-get install mysql
+echo 'rocommunity  jcread 10.1.4.5
+rocommunity  jcread 10.1.4.3
+agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf
-mkdir /data/mysql
+# to see which iface it is, on backup2:
-chown mysql:mysql /data/mysql
-/etc/init.d/mysql stop
-mv /var/lib/mysql/* /data/mysql/
-mv /data/mysql/ib_* /var/lib/mysql/
-vi /etc/mysql/my.cnf
-(change datadir to /data/mysql)
-vi /etc/apparmor.d/usr.sbin.mysqld
-add:
-  /data/mysql/ r,
-  /data/mysql/** rwk,
-Comment out:
-#  /var/lib/mysql/ r,
-#  /var/lib/mysql/** rwk,
-/etc/init.d/apparmor restart
+snmpwalk -v 1 -c jcread 10.1.4.8 interface
-/etc/init.d/mysql start
-tw_cli /c0/u0 set ignoreECC=on
-tw_cli /c0/u0 set storsave=balance
-tw_cli /c0/u0 set cache=on
+echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
-0 * * * /usr/local/sbin/3wraidchk
+echo "bb:x:1984:" >> /etc/group
-</pre>
-= backup2 =
+pwconv
-== Summary ==
+mkdir /home/bb
+chown bb.bb /home/bb
-This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.
+cd ~bb
+scp backup2:/mnt/data4/build/bb/bb-linux.tar .
-* Location: castle, cab 3-7
+tar xf bb-linux.tar
-* OS: FreeBSD 6.1 x86
-* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
-* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
-* Drives:
-**3ware 7500-8:
-***200 GB JBOD (1 x 200G) labeled 0-0
-***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
-***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
-**3ware 7500-8:
-***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
-***700 GB RAID5 (4 x 250G) 1-4 thru 1-7
-All drives MUST be western digital IDE drives. Other brands will not fit.
+cd /home/bb/bbc1.9e-btf/etc
-In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45
+echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
+echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts
-To configure:
+echo "/:90:95
- ifconfig fxp0 link 00:90:27:f9:0a:d9
+/var:90:95
+/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab
-== Services provided ==
-* backup via rsync and nfs
-* samba
-* nfs
-* snmp
-* bigbrother
-== Usage ==
+vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
-* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
+(remove all | SORT xxxx)
-* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
-* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
+chmod +r /var/log/messages
-* this server serves as an archive for exported db data from bwdb and old flow files.
-* isys backs up here
+./bbchkcfg.sh
-* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
+#(y to questions)
-* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
+./bbchkhosts.sh
-** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
+#(ignore ssh errors)
-** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
+cd ../..
-** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems
+chown -R bb .
+su bb
+cd
+cd bbc1.9e-btf/src
-== Cronjobs ==
+#make; make install
-* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)
+cd ..
+./runbb.sh start
+more BBOUT
+(look for errors)
+exit
-== Regular maintenance ==
+vi /etc/rc.local
-*[[Routine_Maintenance#3ware|Check on health]]
+su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
-= backup3 =
-== Summary ==
-This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.
-* Location: i2b, cab 6
+echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports
-* OS: Ubuntu 10.04.1 server amd64
-* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
-* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
-* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port
-== Services provided ==
+/etc/init.d/nfs-kernel-server restart
-* backup via rsync and nfs
-* samba
-* nfs
-* digi realport
-* snmp
-* bigbrother
-== Usage ==
-* all data is stored under /data
-* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
-* this server serves as an archive for exported db data from bwdb and old flow files.
-* inftrastructure machines at i2b back up here
-* customers are nfs-moutned under /data/customers as file-backed loopback devices
-== management scripts ==
+echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd
-* mkbackups
-== Cronjobs ==
-0 * * * /usr/local/sbin/arecaraidchk
-RAID checks
-4 * * * /usr/local/sbin/snapshot_archive
+echo '10.1.4.8                backup1' >> /etc/hosts
-Rotate daily snapshots for infrastructure machine backups
+echo '/dev/sdb1	/data  ext3  rw,noatime  0  0' >> /etc/fstab
-== Regular maintenance ==
-*[[Routine_Maintenance#Areca|Check on RAID health]]
-== Build ==
+to install digi drivers:
-=== BIOS Config ===
+wget http://ftp1.digi.com/support/driver/40002086_n.tgz
-disable quiet boot
+apt-get install linux-image-2.6.24-19-server
+apt-get install linux-source-2.6.24 (not needed?)
+apt-get install linux-headers-2.6.24-19-server
+apt-get install make
+apt-get install gcc
+apt-get install g++
+apt-get install libncurses5-dev
+apt-get install expect
+apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl
-set to last state after power loss
+cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
+./configure
+make all
+make install
+make postinstall
-set date/time to GMT
+/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8
-enable serial console output (baud rate 115200)
+apt-get install mysql
+mkdir /data/mysql
+chown mysql:mysql /data/mysql
+/etc/init.d/mysql stop
+mv /var/lib/mysql/* /data/mysql/
+mv /data/mysql/ib_* /var/lib/mysql/
+vi /etc/mysql/my.cnf
+(change datadir to /data/mysql)
+vi /etc/apparmor.d/usr.sbin.mysqld
+add:
+  /data/mysql/ r,
+  /data/mysql/** rwk,
+Comment out:
+#  /var/lib/mysql/ r,
+#  /var/lib/mysql/** rwk,
+/etc/init.d/apparmor restart
+/etc/init.d/mysql start
+tw_cli /c0/u0 set ignoreECC=on
+tw_cli /c0/u0 set storsave=balance
+tw_cli /c0/u0 set cache=on
-=== Install OS ===
-<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
-G / ext3
-G swap
-~ /data ext4
-Install packages:
+0 * * * /usr/local/sbin/3wraidchk
-openssh
+</pre>
-samba</pre>
-=== DNS and private IP ===
+= backup2 =
- echo "nameserver 69.55.225.225" >> /etc/resolv.conf
+== Summary ==
-Add a 2nd IP to eth0 and setup priv net
+THIS SERVER IS OUT OF SERVICE AND REPLACED BY BACKUP4
-<pre>vi /etc/network/interfaces
-auto eth0
+This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.
-iface eth0 inet static
-        address 69.55.229.4
-        netmask 255.255.255.0
-        network 69.55.229.0
-        broadcast 69.55.229.255
-        gateway 69.55.229.1
-        # dns-* options are implemented by the resolvconf package, if installed
-        dns-nameservers 69.55.229.3 66.181.0.2
-        dns-search johncompanies.com
-auto eth0:1
+* Location: castle, cab 3-7
-iface eth0:1 inet static
+* OS: FreeBSD 6.1 x86
-        address 69.55.231.2
+* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
-        netmask 255.255.255.0
+* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
-        network 69.55.231.0
+* Drives:
-        broadcast 69.55.231.255
+**3ware 7500-8:
+***200 GB JBOD (1 x 200G) labeled 0-0
+***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
+***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
+**3ware 7500-8:
+***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
+***700 GB RAID5 (4 x 250G) 1-4 thru 1-7
-auto eth1
+All drives MUST be western digital IDE drives. Other brands will not fit.
-iface eth1 inet static
-        address 10.1.2.3
-        netmask 255.255.255.0
-        network 10.1.2.0
-        broadcast 10.1.2.255
-</pre>
+In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45
-=== Install packages ===
+To configure:
-<pre>apt-get update
+ ifconfig fxp0 link 00:90:27:f9:0a:d9
-apt-get upgrade
-apt-get install gcc
-apt-get install libssl-dev
-apt-get install libncurses5-dev
-apt-get install cu
-apt-get install unzip
-apt-get install snmp snmpd ntp nfs-kernel-server</pre>
-=== tweak grub, enable serial ===
+== Services provided ==
+* backup via rsync and nfs
+* samba
+* nfs
+* snmp
+* bigbrother
-<pre>vi /etc/default/grub
+== Usage ==
-#GRUB_HIDDEN_TIMEOUT=0
+* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
-GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
+* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
-GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
+* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
-update-grub</pre>
+* this server serves as an archive for exported db data from bwdb and old flow files.
+* isys backs up here
+* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
+* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
+** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
+** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
+** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems
-<pre>echo "start on stopped rc RUNLEVEL=[2345]
+== Cronjobs ==
-stop on runlevel [!2345]
+* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)
-respawn
-exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>
-=== install realport (digi) driver ===
+== Regular maintenance ==
+*[[Routine_Maintenance#3ware|Check on health]]
-give the digi an ip with DgIpServ.exe
+= backup3 =
+== Summary ==
+This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.
-<pre>cd /usr/src/
+* Location: i2b, cab 6
-wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
+* OS: Ubuntu 10.04.1 server amd64
-tar xzf dgrp-1.9.tgz
+* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
-cd dgrp-1.9/
+* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
-./configure
+* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port
-make
-make install
+== Services provided ==
-make postinstall
+* backup via rsync and nfs
-update-rc.d dgrp_daemon defaults</pre>
+* samba
+* nfs
+* digi realport
+* snmp
+* bigbrother
-configure ports:
+== Usage ==
- dgrp_cfg_node init el 10.1.2.10 16
+* all data is stored under /data
+* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
+* this server serves as an archive for exported db data from bwdb and old flow files.
+* inftrastructure machines at i2b back up here
+* customers are nfs-moutned under /data/customers as file-backed loopback devices
-try connecting with:
+== management scripts ==
- cu -l /dev/ttyel00 -s 38400
+* mkbackups
-=== shell, ntp, ssh key, hosts ===
+mkbackup <cid> GB <ip>
-Shell autocompletion search:
+== Cronjobs ==
-<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
+0 * * * /usr/local/sbin/arecaraidchk
-echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>
+RAID checks
-Setup ntp:
+4 * * * /usr/local/sbin/snapshot_archive
-<pre>vi /etc/ntp.conf
+Rotate daily snapshots for infrastructure machine backups
-server 10.1.2.1
-server ntp.ubuntu.com</pre>
-Generate ssh keys:
+== Regular maintenance ==
-<pre>cd /root/
+*[[Routine_Maintenance#Areca|Check on RAID health]]
-ssh-keygen -t dsa</pre>
-Defaults, no password
-Setup hosts:
+== Build ==
-<pre>echo "69.55.230.10 backup2" >> /etc/hosts
-echo "69.55.230.11 backup1" >> /etc/hosts
-echo "10.1.2.4 bwdb2" >> /etc/hosts
-echo "10.1.2.3 backup3" >> /etc/hosts</pre>
-Copy keys to servers where we need passwordless login:
+=== BIOS Config ===
-<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
+disable quiet boot
-cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>
-Setup shell:
+set to last state after power loss
-<pre>vi /root/.bashrc
-(add to bottom)
-alias h='history'
-alias vi='vim'
-alias j='jobs'
-export PS1="[\u@\h \w]# "
-alias dr='screen -dr'
-export EDITOR=vim
-export GREP_OPTIONS='--color=auto'
-export HISTFILESIZE=1000
-alias tip-switch-p20='cu -l ttyel00 -s 9600'
+set date/time to GMT
-alias tip-switch-p21='cu -l ttyel15 -s 9600'
-alias tip-switch-p22='cu -l ttyel14 -s 9600'
-alias tip-switch-p23='cu -l ttyel05 -s 9600'
-alias tip-switch-p24='cu -l ttyel06 -s 9600'
-alias tip-switch-p25='cu -l ttyel09 -s 9600'
-alias tip-switch-p26='cu -l ttyel07 -s 9600'
-alias tip-switch-p27='cu -l ttyel08 -s 9600'
-alias tip-firewall2='cu -l ttyel01 -s 115200'
-alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
-alias tip-backup3='cu -l ttyel04 -s 38400'
-alias tip-bwdb2='cu -l ttyel03 -s 115200'
-alias tip-backup4='cu -l ttyel13 -s 115200'
-alias tip-jail3='cu -l ttyel11 -s 115200'
-Load new shell:
+enable serial console output (baud rate 115200)
- source /root/.bashrc
-Setup snmpd (this is only valid for a server at castle):
+=== Install OS ===
-echo 'rocommunity  jcread 10.1.4.5
+<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
-rocommunity  jcread 10.1.4.3
+G / ext3
-agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf
+G swap
+~ /data ext4
-to see which iface it is, on backup2:
+Install packages:
+openssh
+samba</pre>
-snmpwalk -v 1 -c jcread 10.1.4.8 interface
+=== DNS and private IP ===
-=== nfs ===
+ echo "nameserver 69.55.225.225" >> /etc/resolv.conf
-Allow mounts from private net:
+Add a 2nd IP to eth0 and setup priv net
- echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports
+<pre>vi /etc/network/interfaces
-Restart nfsd:
+auto eth0
- /etc/init.d/nfs-kernel-server restart
+iface eth0 inet static
+        address 69.55.229.4
+        netmask 255.255.255.0
+        network 69.55.229.0
+        broadcast 69.55.229.255
+        gateway 69.55.229.1
+        # dns-* options are implemented by the resolvconf package, if installed
+        dns-nameservers 69.55.229.3 66.181.0.2
+        dns-search johncompanies.com
-=== bb ===
+auto eth0:1
+iface eth0:1 inet static
+        address 69.55.231.2
+        netmask 255.255.255.0
+        network 69.55.231.0
+        broadcast 69.55.231.255
-Add user, group:
+auto eth1
- echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
+iface eth1 inet static
- echo "bb:x:1984:" >> /etc/group
+        address 10.1.2.3
- pwconv
+        netmask 255.255.255.0
+        network 10.1.2.0
+        broadcast 10.1.2.255
-Create home:
+</pre>
- mkdir /home/bb
- chown bb.bb /home/bb
+=== Install packages ===
- cd ~bb
+<pre>apt-get update
+apt-get upgrade
+apt-get install gcc
+apt-get install libssl-dev
+apt-get install libncurses5-dev
+apt-get install cu
+apt-get install unzip
+apt-get install snmp snmpd ntp nfs-kernel-server</pre>
-Copy over and install files:
+=== tweak grub, enable serial ===
-<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
-tar xf bb-linux.tar
-cd /home/bb/bbc1.9e-btf/etc</pre>
-Configure main bb server:
+<pre>vi /etc/default/grub
- echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
+#GRUB_HIDDEN_TIMEOUT=0
- echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts
+GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
+GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
+update-grub</pre>
-Configure low disk alerts:
+<pre>echo "start on stopped rc RUNLEVEL=[2345]
-<pre>echo "/:90:95
+stop on runlevel [!2345]
-/var:90:95
+respawn
-/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>
+exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>
- vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
+=== install realport (digi) driver ===
-(remove all | SORT xxxx since SORT is broken)
- chmod +r /var/log/messages
+give the digi an ip with DgIpServ.exe
- ./bbchkcfg.sh
+<pre>cd /usr/src/
-(y to questions)
+wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
- ./bbchkhosts.sh
+tar xzf dgrp-1.9.tgz
-(ignore ssh errors)
+cd dgrp-1.9/
-<pre>cd ../..
+./configure
-chown -R bb .
+make
-su bb
+make install
-cd
+make postinstall
-cd bbc1.9e-btf/src</pre>
+update-rc.d dgrp_daemon defaults</pre>
-<pre>make; make install
+configure ports:
-cd ..
+ dgrp_cfg_node init el 10.1.2.10 16
-./runbb.sh start
-more BBOUT</pre>
-(look for errors)
- exit
-<pre>vi /etc/rc.local
+try connecting with:
-su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
+ cu -l /dev/ttyel00 -s 38400
-</pre>
-(before the exit 0)
- echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd
+=== shell, ntp, ssh key, hosts ===
-Add f/w rule:
+Shell autocompletion search:
- ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984
+<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
+echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>
-<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
+Setup ntp:
-DFWARN=199
+<pre>vi /etc/ntp.conf
-DFPANIC=199</pre>
+server 10.1.2.1
+server ntp.ubuntu.com</pre>
+Generate ssh keys:
+<pre>cd /root/
+ssh-keygen -t dsa</pre>
+Defaults, no password
-=== raid check ===
+Setup hosts:
+<pre>echo "69.55.230.10 backup2" >> /etc/hosts
+echo "69.55.230.11 backup1" >> /etc/hosts
+echo "10.1.2.4 bwdb2" >> /etc/hosts
+echo "10.1.2.3 backup3" >> /etc/hosts</pre>
-==== 3ware ====
+Copy keys to servers where we need passwordless login:
-<pre>
+<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
-scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
+cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>
-scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
-scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
-vi /usr/local/sbin/checkraid.sh
-:%s/c0/c2/g
-crontab -e
+Setup shell:
-0 * * * /usr/local/sbin/3wraidchk</pre>
+<pre>vi /root/.bashrc
+(add to bottom)
+alias h='history'
+alias vi='vim'
+alias j='jobs'
+export PS1="[\u@\h \w]# "
+alias dr='screen -dr'
+export EDITOR=vim
+export GREP_OPTIONS='--color=auto'
+export HISTFILESIZE=1000
-==== areca ====
+alias tip-switch-p20='cu -l ttyel00 -s 9600'
-<pre>
+alias tip-switch-p21='cu -l ttyel15 -s 9600'
-cd /tmp
+alias tip-switch-p22='cu -l ttyel14 -s 9600'
-wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
+alias tip-switch-p23='cu -l ttyel05 -s 9600'
-unzip linuxcli_V1.10.0_120815.zip
+alias tip-switch-p24='cu -l ttyel06 -s 9600'
-cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
+alias tip-switch-p25='cu -l ttyel09 -s 9600'
-chmod 0700 /usr/local/sbin/cli64
+alias tip-switch-p26='cu -l ttyel07 -s 9600'
-cli64 rsf info
+alias tip-switch-p27='cu -l ttyel08 -s 9600'
-</pre>
+alias tip-firewall2='cu -l ttyel01 -s 115200'
+alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
+alias tip-backup3='cu -l ttyel04 -s 38400'
+alias tip-bwdb2='cu -l ttyel03 -s 115200'
+alias tip-backup4='cu -l ttyel13 -s 115200'
+alias tip-jail3='cu -l ttyel11 -s 115200'
-<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
+Load new shell:
-scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin
+ source /root/.bashrc
-crontab -e
+Setup snmpd (this is only valid for a server at castle):
-0 * * * /usr/local/sbin/arecaraidchk
+echo 'rocommunity  jcread 10.1.4.5
-</pre>
+rocommunity  jcread 10.1.4.3
+agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf
-<pre>cat > /root/verify.sh
+to see which iface it is, on backup2:
-cli64 vsf info
-cli64 rsf info
-cli64 disk info
-cli64 event info
-echo press enter when ready to run verify ; read x
-cli64 vsf check vol=1
+snmpwalk -v 1 -c jcread 10.1.4.8 interface
-</pre>
-=== misc binaries ===
+=== nfs ===
- scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
+Allow mounts from private net:
-  vi /usr/local/sbin/snapshot_archive
+  echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports
-(remove entries)
- crontab -e
+Restart nfsd:
-4 * * * /usr/local/sbin/snapshot_archive
+  /etc/init.d/nfs-kernel-server restart
- scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
+=== bb ===
- scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone
-Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:
+Add user, group:
+ echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
+ echo "bb:x:1984:" >> /etc/group
+ pwconv
-<pre>
+Create home:
-cat > /bin/mail
+ mkdir /home/bb
-#!/usr/bin/perl
+ chown bb.bb /home/bb
-use strict;
+ cd ~bb
-use warnings;
-use lib '/usr/local/sbin';
+Copy over and install files:
-use Sendmail qw(sendmail);
+<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
+tar xf bb-linux.tar
+cd /home/bb/bbc1.9e-btf/etc</pre>
-my $sub = $ARGV[1];
+Configure main bb server:
-my $to = $ARGV[2];
+ echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
+ echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts
+Configure low disk alerts:
+<pre>echo "/:90:95
+/var:90:95
+/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>
-my %mail = (
+ vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
-   To      => $to,
+(remove all | SORT xxxx since SORT is broken)
-   From    => $to,
-   Subject => $sub,
-   Message => '',
-   smtp    => 'mail.johncompanies.com'
-);
-sendmail(%mail) || print "Error: $Sendmail::error";
-</pre>
+ chmod +r /var/log/messages
-  chmod 0700 /bin/mail
+  ./bbchkcfg.sh
+(y to questions)
-=== mkbackup ===
+ ./bbchkhosts.sh
+(ignore ssh errors)
+<pre>cd ../..
+chown -R bb .
+su bb
+cd
+cd bbc1.9e-btf/src</pre>
+<pre>make; make install
+cd ..
+./runbb.sh start
+more BBOUT</pre>
+(look for errors)
+ exit
- mkdir /data/customers
+<pre>vi /etc/rc.local
+su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
+</pre>
+(before the exit 0)
-<pre>cat > /usr/local/sbin/mkbackup
+ echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd
-#!/bin/sh
-if test $1; then
+Add f/w rule:
-  cid=$1
+ ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984
-else
-  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
-  exit
-fi
-if test $2; then
+<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
-  gb=$2
+DFWARN=199
-else
+DFPANIC=199</pre>
-  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
-  exit
-fi
-if test $3; then
-  ip=$3
-else
-  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
-  exit
-fi
+=== raid check ===
-if test -e /data/customers/${cid}-file; then
+==== 3ware ====
-  echo "ERROR: /data/customers/${cid}-file exists"
+<pre>
-  exit
+scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
-else
+scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
-  echo "touch /data/customers/${cid}-file"
+scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
-  touch /data/customers/${cid}-file
+vi /usr/local/sbin/checkraid.sh
-  count=`echo $gb|awk '{print $1*1000}'`
+:%s/c0/c2/g
-  echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
-  dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
-  echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
-  /sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
-fi
-if test -e /data/customers/$cid; then
+crontab -e
-  echo "ERROR: /data/customers/$cid exists"
+0 * * * /usr/local/sbin/3wraidchk</pre>
-  exit
-else
-  echo "mkdir /data/customers/${cid}"
-  mkdir /data/customers/${cid}
-  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
-  mount -o loop /data/customers/${cid}-file /data/customers/$cid
-  df -h /data/customers/$cid
-  echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
+==== areca ====
-  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
+<pre>
-  echo "" >> /etc/nfs_backup_mounts.sh
+cd /tmp
+wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
+unzip linuxcli_V1.10.0_120815.zip
+cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
+chmod 0700 /usr/local/sbin/cli64
+cli64 rsf info
+</pre>
-  echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
+<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
-  /etc/init.d/nfs-kernel-server restart
+scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin
-  tail /var/log/messages
-fi</pre>
- chmod 0700 /usr/local/sbin/mkbackup
+crontab -e
+0 * * * /usr/local/sbin/arecaraidchk
+</pre>
- vi /etc/rc.local
+<pre>cat > /root/verify.sh
-add:
+cli64 vsf info
- /etc/nfs_backup_mounts.sh
+cli64 rsf info
+cli64 disk info
+cli64 event info
+echo press enter when ready to run verify ; read x
-=== samba ===
+cli64 vsf check vol=1
+</pre>
- apt-get install samba
+=== misc binaries ===
-  vi /etc/samba/smb.conf
+ scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
+  vi /usr/local/sbin/snapshot_archive
+(remove entries)
-; comment out any mounts, add:
+ crontab -e
+4 * * * /usr/local/sbin/snapshot_archive
-<pre>[data]
+ scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
-   read only = yes
+ scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone
-   locking = no
-   path = /data/iso
-   guest ok = yes</pre>
- /etc/init.d/smbd restart
+Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:
- mkdir /data/iso
+<pre>
+cat > /bin/mail
+#!/usr/bin/perl
+use strict;
+use warnings;
-Bring over some stuff from backup2
+use lib '/usr/local/sbin';
+use Sendmail qw(sendmail);
-<pre>cd /data/iso
+my $sub = $ARGV[1];
-scp backup2:/d2/iso/3wfirmware.iso .
+my $to = $ARGV[2];
-scp backup2:/d2/iso/MD5SUMS .
-scp backup2:/d2/iso/bootimg.iso .
-scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
-scp backup2:/d2/iso/win98bootcd.iso .
-scp backup2:/d2/iso/acronis_bootdisk.iso .
-scp backup2:/d2/iso/memtest86-3.2.iso .</pre>
-=== Moving from one server to another ===
-Here are the steps you would take to move settings and data from one server to a new backup server:
+my %mail = (
+   To      => $to,
+   From    => $to,
+   Subject => $sub,
+   Message => '',
+   smtp    => 'mail.johncompanies.com'
+);
+sendmail(%mail) || print "Error: $Sendmail::error";
-* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
+</pre>
- rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
-after umounting all the customers, copy over the (empty) directories separately:
- for f in `find .  -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done
-* copy mount script
+  chmod 0700 /bin/mail
-  [root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh
-* copy rc.local
+=== mkbackup ===
- [root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local
-* copy /etc/exports
+  mkdir /data/customers
-  [root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports
-* edit /etc/hostname on both machines (set current to oldbackup3)
+<pre>cat > /usr/local/sbin/mkbackup
+#!/bin/sh
-* edit /etc/network/interfaces (swap IPs).
+if test $1; then
+  cid=$1
+else
+  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
+  exit
+fi
-* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
+if test $2; then
-  chmod 000 /etc/nfs_backup_mounts.sh
+  gb=$2
+else
+  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
+  exit
+fi
-* reboot both servers @ same time
+if test $3; then
+  ip=$3
+else
+  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
+  exit
+fi
-* check everything out
-* run /etc/nfs_backup_mounts.sh on new server
+if test -e /data/customers/${cid}-file; then
+  echo "ERROR: /data/customers/${cid}-file exists"
+  exit
+else
+  echo "touch /data/customers/${cid}-file"
+  touch /data/customers/${cid}-file
+  count=`echo $gb|awk '{print $1*1000}'`
+  echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
+  dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
+  echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
+  /sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
+fi
-* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
+if test -e /data/customers/$cid; then
- vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
+  echo "ERROR: /data/customers/$cid exists"
+  exit
+else
+  echo "mkdir /data/customers/${cid}"
+  mkdir /data/customers/${cid}
+  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
+  mount -o loop /data/customers/${cid}-file /data/customers/$cid
+  df -h /data/customers/$cid
-= console =
+  echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
+  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
+  echo "" >> /etc/nfs_backup_mounts.sh
-== Summary ==
+  echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
-This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.
+  /etc/init.d/nfs-kernel-server restart
+  tail /var/log/messages
+fi</pre>
-* Location: castle, cab 3-8
+ chmod 0700 /usr/local/sbin/mkbackup
-* OS: SunOS 5.8 (solaris)
-* Networking: Priv IP: 10.1.4.4
-* Hardware: Sun Netra
+ vi /etc/rc.local
+add:
+ /etc/nfs_backup_mounts.sh
-To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>
+=== samba ===
-i.e.
+  apt-get install samba
-  tip switch-p1
- tip jail1
-== Configuring digi/ports ==
+ vi /etc/samba/smb.conf
-=== /etc/remote ===
+; comment out any mounts, add:
-This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.
-We have 2 digi's at castle we connect to:
+<pre>[data]
+   read only = yes
+   locking = no
+   path = /data/iso
+   guest ok = yes</pre>
-  #3-7 10.1.4.10
+  /etc/init.d/smbd restart
- virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
- virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
-and
+ mkdir /data/iso
- #3-6 10.1.4.11
+Bring over some stuff from backup2
- jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
- jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
-The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)
+<pre>cd /data/iso
+scp backup2:/d2/iso/3wfirmware.iso .
-=== drpadmin ===
+scp backup2:/d2/iso/MD5SUMS .
-The tool you use to configure a device to a digi box is drpadmin:
+scp backup2:/d2/iso/bootimg.iso .
+scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
+scp backup2:/d2/iso/win98bootcd.iso .
+scp backup2:/d2/iso/acronis_bootdisk.iso .
+scp backup2:/d2/iso/memtest86-3.2.iso .</pre>
-<pre>bash-2.03$ su
+=== Moving from one server to another ===
-Password:
-# drpadmin
-Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
+Here are the steps you would take to move settings and data from one server to a new backup server:
-       10.1.4.10       32      CO      771     never   1027
-       10.1.4.11       32      CP      771     never   1027
-       65.116.11.2     8       el      771     never   1027
-Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>
+* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
+ rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
+after umounting all the customers, copy over the (empty) directories separately:
+ for f in `find .  -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done
-Use those commands above to modify the devices available.
+* copy mount script
+ [root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh
-== Switching IP/hostname ==
+* copy rc.local
+ [root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local
-Edit:
+* copy /etc/exports
- /etc/defaultrouter
+  [root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports
-  /etc/hosts
- /etc/hostname.hme0
- /etc/nodename
- Maybe needed to run: # ifconfig hme0 10.1.4.4 up
-= devweb =
+* edit /etc/hostname on both machines (set current to oldbackup3)
-We do web development on devweb.johncompanies.com
+* edit /etc/network/interfaces (swap IPs).
-Currently this is a jail running on jail17 / 69.55.230.8
+* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
+ chmod 000 /etc/nfs_backup_mounts.sh
-If the jail is restarted, you will need to manually restart the web service with:
+* reboot both servers @ same time
- httpsdctl restart
-All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].
+* check everything out
+* run /etc/nfs_backup_mounts.sh on new server
-= firewall2 =
+* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
+ vi /usr/local/www/mgmt/mrtg/mrtg1.cfg
+= backup4 =
 == Summary ==
+This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs.  Only FreeBSD virt customers directly accesses this server to perform their backups.
+* Location: castle, cab 3-7
+* OS: FreeNAS 9.3 (FreeBSD 9.3)
+* Networking: Priv IP: 10.1.2.9/24 AND 10.1.7.9/24,  Pub IPs: 69.55.230.6/24
+* Hardware: JC-08014
+            Intel S5000VSA Motherboard
+x Intel Xeon E5410  @ 2.33GHz CPU
+ware 9690SA-8I RAID Card w BBU
+GB RAM
+            Dual power supply.
+* Drives: 7 TB (6 x 2TB) ZFS RAIDZ2 array running on JBOD
+128 GB SSD system drive and 6 drive SATA bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap.
-This machine is the primary (only) firewall for the entire network at i2b.
+* GUI management at http://backup4.johncompanies.com
-* Location: i2b, cab 6
+== Services provided ==
-* OS: FreeBSD 6.4 x86
+* backup via rsync and nfs
-* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation
+* samba
+* nfs
+* snmp?
+* bigbrother?
-* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+== Usage ==
-* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.
+* all data is stored under /data
+* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>  ??
+* this server serves as an archive for exported db data from bwdb and old flow files. ??
+* customers are nfs-moutned under /data/users (/mnt/zfs/users) as zfs ?
-== Services Provided ==
+== management scripts ==
-* firewall (ipfw)
+* mkbackups?
-* bigbrother
-== Firewall Rule Configuration ==
+mkbackup <cid> GB <ip>
-See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.
+== Cronjobs ==
+0 * * * /usr/local/sbin/arecaraidchk
+RAID checks ?
-== Disaster Recovery ==
+4 * * * /usr/local/sbin/snapshot_archive
+Rotate daily snapshots for infrastructure machine backups
-TODO: need backup f/w and instructions on how to move cables.
+15 * * * /usr/local/sbin/snapshot_rotate
+Rotate daily snapshots for customer machine backups
-Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)
+== Regular maintenance ==
+*[[Routine_Maintenance#A|Check on RAID health]]
-TODO
+== Build ==
-Here's the config on the live firewall:
+= console =
-<pre>kern_securelevel_enable="NO"
+== Summary ==
-portmap_enable="NO"
+This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.
-sendmail_enable="NO"
-usbd_enable="YES"
-gateway_enable="YES"
-xntpd_enable="YES"
+OBSOLETE
-nfs_client_enable="YES"
+* Location: castle, cab 3-8
-nfs_reserved_port_only="YES"
+* OS: SunOS 5.8 (solaris)
-inetd_enable="YES"
+* Networking: Priv IP: 10.1.4.4
-inetd_flags="-wW -a 10.1.2.2"
+* Hardware: Sun Netra
-fsck_y_enable="YES"
-background_fsck="NO"
-defaultrouter="66.181.18.2"
+To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>
-hostname="firewall2.johncompanies.com"
-ifconfig_bge0="inet 66.181.18.3  netmask 255.255.255.224"
-ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
-ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
-ifconfig_bge1_alias1="inet 65.50.228.1  netmask 255.255.255.0"
-ifconfig_bge1_alias2="inet 65.50.229.1  netmask 255.255.255.0"
-ifconfig_bge1_alias3="inet 65.50.230.1  netmask 255.255.255.0"
-ifconfig_bge1_alias4="inet 65.50.231.1  netmask 255.255.255.0"
-ifconfig_bge1_alias5="inet 65.50.232.1  netmask 255.255.255.0"
-ifconfig_bge1_alias6="inet 65.50.233.1  netmask 255.255.255.0"
-ifconfig_bge1_alias7="inet 65.50.234.1  netmask 255.255.255.0"
-ifconfig_bge1_alias8="inet 65.50.235.1  netmask 255.255.255.0"
-ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
-sshd_enable="YES"
-usbd_enable="YES"
-</pre>
+SSH WORK ~2021
+ssh user@console.johncompanies.com <br>
-== Cronjobs ==
+PW: 674*****
-3 * * * /usr/local/etc/rsync.backup
-Backup to backup3
-0 1 * * /sbin/ipfw zero
+i.e.
-0 1 * * /sbin/ipfw del 3 4 5
+  tip switch-p1
-Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).
+  tip jail1
-23 30 * * /sbin/ipfw show > /tmp/ipfw_count
+== Configuring digi/ports ==
-0 30 * * /sbin/ipfw show > /tmp/ipfw_count
-Capture counts periodically
-3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
+=== /etc/remote ===
-This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.
+This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.
- */5 * * * * /usr/local/sbin/lsiraidchk
+We have 2 digi's at castle we connect to:
-Checking the health of the RAID array
+ #3-7 10.1.4.10
+ virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
+ virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
-== DOS attacks ==
+and
-See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.
+ #3-6 10.1.4.11
+ jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
+ jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
+The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)
-== build ==
+=== drpadmin ===
+The tool you use to configure a device to a digi box is drpadmin:
-<pre>partition map:
+<pre>bash-2.03$ su
-/ 58g
+Password:
-swap 4g
+# drpadmin
-/var 512m
-/tmp 512m
-/usr 5.5g
-. edit /etc/make.conf
+Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
-echo "WITHOUT_X11=yes \
+       10.1.4.10       32      CO      771     never   1027
-KERNCONF=firewall2 \
+       10.1.4.11       32      CP      771     never   1027
-BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
+       65.116.11.2     8       el      771     never   1027
-. add settings to /boot/loader.conf and /boot.config
+Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>
-echo "-Dh" >> /boot.config
+Use those commands above to modify the devices available.
-echo 'console="comconsole,vidconsole" \
+== Switching IP/hostname ==
-boot_multicons="YES" \
-boot_serial="YES" \
-comconsole_speed="115200"' >> /boot/loader.conf
+Edit:
+ /etc/defaultrouter
+ /etc/hosts
+ /etc/hostname.hme0
+ /etc/nodename
+ Maybe needed to run: # ifconfig hme0 10.1.4.4 up
-. turn off all ttyv's except 0 and 1 in /etc/ttys
+= devweb =
-also turn on ttyd0, change type to vt100:
-vi /etc/ttys
-ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
-ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
-ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
-ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
-ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
-ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
-# Serial terminals
-# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
-ttyd0   "/usr/libexec/getty std.9600"   vt100   on  secure
-kill -1 1
+We do web development on devweb.johncompanies.com
-on console server:
+Currently this is a jail running on jail17 / 69.55.230.8
-vi /etc/remote
-(rename port to jail8 depending on where and which digi plugged into)
-test serial console
+If the jail is restarted, you will need to manually restart the web service with:
+ httpsdctl restart
-. populate hosts
+All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].
-echo "69.55.230.10 backup2" >> /etc/hosts
-echo "69.55.230.11 backup1" >> /etc/hosts
-echo "10.1.2.3 backup3" >> /etc/hosts
-. put key in authorized_keys on backup3
-cd
-ssh-keygen -t dsa -b 1024
-(default location, leave password blank)
-Punch a hole in firewall1 to allow traffic to backup servers @ castle:
+= firewall2 =
-ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
+== Summary ==
-ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22
-cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
+This machine is the primary firewall for the entire network at i2b.   firewall3 is a hot standby replacement for
-cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
+firewall2.  Both firewall2 and firewall3 should not be connected at the same time since they use the same internal
-cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
+and external IP addresses.
-confirm that you can ssh to backup3 and backup 2 without getting a login prompt
+* Location: i2b, cab 6
+* OS: FreeBSD 6.4 x86
+* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation
-ssh backup3 hostname
+* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.
-ssh backup2 hostname
+== Services Provided ==
+* firewall (ipfw)
+* bigbrother for customer machines
-ssh backup1 hostname
+== Firewall Rule Configuration ==
+See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.
-. edit root's path and login script:
+== Disaster Recovery ==
-vi /root/.cshrc
-Change alias entries (add G):
+TODO: need backup f/w and instructions on how to move cables.
-alias la        ls -aG
-alias lf        ls -FAG
-alias ll        ls -lAG
-alias ls        ls -AG
-alias mbm       mb mount
-alias mbu       mb umount
-and alter the prompt, set the following:
+Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)
-set prompt = "`/bin/hostname -s` %/# "
-. install cvsup
+TODO
-cd /usr/ports/net/cvsup-without-gui
-make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
-. get latest sources for this release:
+Here's the config on the live firewall:
-cd /usr/src
-echo "*default host=cvsup4.freebsd.org\
-*default base=/usr\
-*default prefix=/usr\
-*default release=cvs tag=RELENG_6_4\
-*default delete use-rel-suffix\
-*default compress\
-src-all" > sup
-cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
+<pre>kern_securelevel_enable="NO"
+portmap_enable="NO"
+sendmail_enable="NO"
+usbd_enable="YES"
+gateway_enable="YES"
-. configure new kernel.
+xntpd_enable="YES"
-cd /usr/src/sys/i386/conf
-scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2
-. build, install kernel and world
-cd /boot
-mv kernel kernel.GENERIC
-cd kernel.GENERIC
-cd /usr/src
-make buildkernel installkernel
-make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
-(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
-make installworld
-(2450: 3min, supermicro: 1min, 2950: :34)
-mergemaster -i
-. populate /etc/rc.conf with IPs and NFS settings
-vi /etc/rc.conf
-kern_securelevel_enable="NO"
-portmap_enable="NO"
-sendmail_enable="NO"
-usbd_enable="YES"
-gateway_enable="YES"
-xntpd_enable="YES"
 nfs_client_enable="YES"
 nfs_reserved_port_only="YES"
@@ Line 4,136: / Line 4,394: @@
 inetd_flags="-wW -a 10.1.2.2"
-ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
 fsck_y_enable="YES"
 background_fsck="NO"
@@ Line 4,144: / Line 4,401: @@
 ifconfig_bge0="inet 66.181.18.3  netmask 255.255.255.224"
 ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
+ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
+ifconfig_bge1_alias1="inet 65.50.228.1  netmask 255.255.255.0"
+ifconfig_bge1_alias2="inet 65.50.229.1  netmask 255.255.255.0"
+ifconfig_bge1_alias3="inet 65.50.230.1  netmask 255.255.255.0"
+ifconfig_bge1_alias4="inet 65.50.231.1  netmask 255.255.255.0"
+ifconfig_bge1_alias5="inet 65.50.232.1  netmask 255.255.255.0"
+ifconfig_bge1_alias6="inet 65.50.233.1  netmask 255.255.255.0"
+ifconfig_bge1_alias7="inet 65.50.234.1  netmask 255.255.255.0"
+ifconfig_bge1_alias8="inet 65.50.235.1  netmask 255.255.255.0"
 ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
 sshd_enable="YES"
 usbd_enable="YES"
+</pre>
-. reboot. Confirm new kernel is loaded
-uname -a
+== Cronjobs ==
+3 * * * /usr/local/etc/rsync.backup
+Backup to backup3
-. update ports:
+0 1 * * /sbin/ipfw zero
-cd /usr/ports
+0 1 * * /sbin/ipfw del 3 4 5
-echo "*default host=cvsup4.FreeBSD.org\
+Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).
-*default base=/usr\
-*default prefix=/usr\
-*default release=cvs tag=RELENG_6_4\
-*default delete use-rel-suffix\
-*default compress\
-ports-all tag=." > sup
-cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
+23 30 * * /sbin/ipfw show > /tmp/ipfw_count
+0 30 * * /sbin/ipfw show > /tmp/ipfw_count
+Capture counts periodically
-. Install raid mgmt tool
+3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
+This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.
-# linux base
+ */5 * * * * /usr/local/sbin/lsiraidchk
-cd /usr/ports/devel/libtool22
+Checking the health of the RAID array
-make install base
-cd /usr/ports/emulators/linux_base-fc4
-make install clean
-#linux-megamgr-5.20
+== DOS attacks ==
-cd /usr/ports/sysutils/linux-megamgr
-make install clean
-# megarc-1.51
+See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.
-cd /usr/ports/sysutils/megarc
-make install clean
-Test:
-rehash; megarc -ldInfo -a0 -l0
-. install rsync from ports
+== build ==
-cd /usr/ports/net/rsync
-make install clean
-choose default options
+<pre>partition map:
+/ 58g
-. install bb client
+swap 4g
-adduser
+/var 512m
-Username: bb
+/tmp 512m
-Full name: bb
+/usr 5.5g
-Uid (Leave empty for default): 1984
-Login group [bb]:
-Login group is bb. Invite bb into other groups? []:
-Login class [default]:
-Shell (sh csh tcsh nologin) [sh]:
-Home directory [/home/bb]:
-Use password-based authentication? [yes]:
-Use an empty password? (yes/no) [no]:
-Use a random password? (yes/no) [no]: yes
-Lock out the account after creation? [no]:
-Username   : bb
-Password   : <random>
-Full Name  : bb
-Uid        : 1984
-Class      :
-Groups     : bb
-Home       : /home/bb
-Shell      : /bin/sh
-Locked     : no
-OK? (yes/no): yes
-cd /usr/home/bb
+. edit /etc/make.conf
-scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
+echo "WITHOUT_X11=yes \
-tar xvf bb-freebsd.tar
+KERNCONF=firewall2 \
+BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
-edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
+. add settings to /boot/loader.conf and /boot.config
-echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
-.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
-vi /home/bb/bbc1.9e-btf/ext/openfiles
+echo "-Dh" >> /boot.config
-MACHINE="firewall2,johncompanies,com"      # HAS TO BE IN A,B,C FORM
-cd /usr/home/bb/bbc1.9e-btf/etc
+echo 'console="comconsole,vidconsole" \
-./bbchkcfg.sh
+boot_multicons="YES" \
-(y to questions)
+boot_serial="YES" \
-./bbchkhosts.sh
+comconsole_speed="115200"' >> /boot/loader.conf
-(ignore ssh errors)
-cd ../..
-chown -R bb .
-su bb
-cd
-cd bbc1.9e-btf/src
-make; make install
-cd ..
-vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
-        $1 $TOPARGS > $BBTMP/TOP.$$
-#        /usr/local/jail/bin/jtop > $BBTMP/TOP.$$
-./runbb.sh start
+. turn off all ttyv's except 0 and 1 in /etc/ttys
-more BBOUT
+also turn on ttyd0, change type to vt100:
-(look for errors)
+vi /etc/ttys
-exit
-echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
+ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
-chmod +x /usr/local/etc/rc.d/bb.sh
+ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
+# Serial terminals
+# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
+ttyd0   "/usr/libexec/getty std.9600"   vt100   on  secure
-Punch a hole in the firewall to allow it to communicate with bb monitor:
+kill -1 1
-ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2
+on console server:
+vi /etc/remote
+(rename port to jail8 depending on where and which digi plugged into)
+test serial console
-. configure bb on mail:
+. populate hosts
-vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
+echo "69.55.230.10 backup2" >> /etc/hosts
-.181.18.3 firewall2.johncompanies.com # ssh
+echo "69.55.230.11 backup1" >> /etc/hosts
+echo "10.1.2.3 backup3" >> /etc/hosts
-su bb
+. put key in authorized_keys on backup3
 cd
-bbsrc/bb/runbb.sh restart ; exit
+ssh-keygen -t dsa -b 1024
+(default location, leave password blank)
-. configure ntp
+Punch a hole in firewall1 to allow traffic to backup servers @ castle:
-echo "server 10.1.2.1" > /etc/ntp.conf
-/usr/sbin/ntpd -p /var/run/ntpd.pid
+ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
-sleep 2; ntpq -p
+ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22
-(confirm it’s able to reach our time server)
-echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
+cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
-chmod 0700 /usr/local/etc/rc.d/ntp.sh
+cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
+cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
-. fwd and reverse lookups on ns1c
+confirm that you can ssh to backup3 and backup 2 without getting a login prompt
-vr johncompanies.com
- (edit the PTR too)
+ssh backup3 hostname
-. setup backups
+ssh backup2 hostname
-echo '#\!/bin/sh\
-backupdir=/data/firewall2/current\
-\
-## ENTRY /etc ' > /usr/local/etc/backup.config
-on backup3:
+ssh backup1 hostname
-setup backup dirs:
-ssh backup3 mkdir -p /data/firewall2/current
-on backup3, add the system to
-vi /usr/local/sbin/snapshot_archive
-scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
+. edit root's path and login script:
-vi /usr/local/etc/rsync.backup
+vi /root/.cshrc
-backup1 > backup3
-crontab -e
+Change alias entries (add G):
-0 * * * /usr/local/etc/rsync.backup
+alias la        ls -aG
+alias lf        ls -FAG
+alias ll        ls -lAG
+alias ls        ls -AG
+alias mbm       mb mount
+alias mbu       mb umount
+and alter the prompt, set the following:
+set prompt = "`/bin/hostname -s` %/# "
-. mkdir /root/logs
+. install cvsup
+cd /usr/ports/net/cvsup-without-gui
+make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null
-. edit sshd_config for security
+. get latest sources for this release:
-vi /etc/ssh/sshd_config
+cd /usr/src
-ListenAddress 66.181.18.3
+echo "*default host=cvsup4.freebsd.org\
-ListenAddress 10.1.2.1
+*default base=/usr\
+*default prefix=/usr\
+*default release=cvs tag=RELENG_6_4\
+*default delete use-rel-suffix\
+*default compress\
+src-all" > sup
-kill -1 `cat /var/run/sshd.pid`
+cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null
-. raid chk
+. configure new kernel.
-cat > /usr/local/sbin/lsiraidchk
+cd /usr/src/sys/i386/conf
-#!/usr/bin/perl
+scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2
-my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;
+. build, install kernel and world
-foreach (@out) {
+cd /boot
-    if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
-#print $_;
-}
-. add crontab entries
+mv kernel kernel.GENERIC
-crontab -e
+cd kernel.GENERIC
-3 * * * /usr/local/etc/rsync.backup
+cd /usr/src
-0 1 * * /sbin/ipfw zero
+make buildkernel installkernel
-0 1 * * /sbin/ipfw del 3 4 5
-23 30 * * /sbin/ipfw show > /tmp/ipfw_count
-0 30 * * /sbin/ipfw show > /tmp/ipfw_count
-3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
-*/5 * * * * /usr/local/sbin/lsiraidchk
-#10 0 * * * rm /var/spool/clientmqueue/*
+make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
+(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
+make installworld
+(2450: 3min, supermicro: 1min, 2950: :34)
+mergemaster -i
-scp /etc/makefwrules.pl user@64.163.14.48:~
+. populate /etc/rc.conf with IPs and NFS settings
-scp /etc/makepiperules.pl user@64.163.14.48:~
+vi /etc/rc.conf
-mv /home/user/makefwrules.pl /etc
-mv /home/user/makepiperules.pl /etc
-touch /etc/firewall.sh
-mkdir /etc/oldrules/
-other binaries
+kern_securelevel_enable="NO"
+portmap_enable="NO"
-scp /usr/local/bin/rulemaker user@64.163.14.48:~
+sendmail_enable="NO"
-mv ~user/rulemaker /usr/local/sbin
+usbd_enable="YES"
-scp ~user/Sendmail.pm user@64.163.14.48:~
+gateway_enable="YES"
-scp ~user/doswatch.pl user@64.163.14.48:~
-Setup basic ruleset
+xntpd_enable="YES"
+nfs_client_enable="YES"
+nfs_reserved_port_only="YES"
+inetd_enable="YES"
+inetd_flags="-wW -a 10.1.2.2"
-ipfw add 00009 count udp from any to any
+ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
-ipfw add 00010 allow tcp from any to any established
+fsck_y_enable="YES"
-ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
+background_fsck="NO"
-ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
-ipfw add 00012 deny tcp from any to any tcpflags syn,fin
-ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
-ipfw add 00012 allow icmp from any to any
-ipfw add 00014 deny tcp from any to any dst-port 135
-ipfw add 00150 skipto 65535 ip from any to any via em1 in
-IPKVM3:
+defaultrouter="66.181.18.2"
-allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
+hostname="firewall2.johncompanies.com"
-deny ip from any to 69.55.230.10 dst-port 139
+ifconfig_bge0="inet 66.181.18.3  netmask 255.255.255.224"
-</pre>
+ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
+ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
+sshd_enable="YES"
+usbd_enable="YES"
-= wiki =
+. reboot. Confirm new kernel is loaded
-The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8
+uname -a
-The backup wiki lives on virt13 in CT 5 / 69.55.230.18
+. update ports:
+cd /usr/ports
+echo "*default host=cvsup4.FreeBSD.org\
+*default base=/usr\
+*default prefix=/usr\
+*default release=cvs tag=RELENG_6_4\
+*default delete use-rel-suffix\
+*default compress\
+ports-all tag=." > sup
-== Setup jail ==
+cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null
-<pre>
+. Install raid mgmt tool
-mkdir /mnt/data1/wiki
-cd /usr/src
-make installworld DESTDIR=/mnt/data1/wiki
-cd etc
-make distribution DESTDIR=/mnt/data1/wiki
-mount -t devfs devfs /mnt/data1/wiki/dev
+# linux base
-devfs -m /mnt/data1/wiki/dev rule -s 3 applyset
+cd /usr/ports/devel/libtool22
+make install base
-cd /mnt/data1/wiki
+cd /usr/ports/emulators/linux_base-fc4
+make install clean
-ln -sf dev/null kernel
+#linux-megamgr-5.20
+cd /usr/ports/sysutils/linux-megamgr
+make install clean
-scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki/sbin
+# megarc-1.51
+cd /usr/ports/sysutils/megarc
+make install clean
-jail /mnt/data1/wiki wiki.johncompanies.com 69.55.229.8 /bin/sh
+Test:
-csh
+rehash; megarc -ldInfo -a0 -l0
-touch /etc/fstab
+. install rsync from ports
-echo 'network_interfaces=""\
+cd /usr/ports/net/rsync
-hostname="wiki.johncompanies.com"\
+make install clean
-kern_securelevel_enable="NO"\
-sendmail_enable="YES"\
-sshd_enable="YES"' > /etc/rc.conf
-echo "nameserver 69.55.229.3\
+choose default options
-nameserver 69.55.225.225" >> /etc/resolv.conf
-vi /etc/crontab
+. install bb client
-(remove the adjkerntz lines )
-vi /etc/periodic/security/100.chksetuid
-replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
- with: MP='/' (use single quotes)
-mkdir -p /usr/compat/linux/dev
 adduser
+Username: bb
-Username: user
+Full name: bb
-Full name: user
+Uid (Leave empty for default): 1984
-Uid (Leave empty for default):
+Login group [bb]:
-Login group [user]:
+Login group is bb. Invite bb into other groups? []:
-Login group is user. Invite user into other groups? []: wheel
 Login class [default]:
 Shell (sh csh tcsh nologin) [sh]:
-Home directory [/home/user]:
+Home directory [/home/bb]:
+Use password-based authentication? [yes]:
+Use an empty password? (yes/no) [no]:
+Use a random password? (yes/no) [no]: yes
+Lock out the account after creation? [no]:
+Username   : bb
+Password   : <random>
+Full Name  : bb
+Uid        : 1984
+Class      :
+Groups     : bb
+Home       : /home/bb
+Shell      : /bin/sh
+Locked     : no
+OK? (yes/no): yes
+cd /usr/home/bb
+scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
+tar xvf bb-freebsd.tar
+edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
+echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
+.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
+vi /home/bb/bbc1.9e-btf/ext/openfiles
+MACHINE="firewall2,johncompanies,com"      # HAS TO BE IN A,B,C FORM
+cd /usr/home/bb/bbc1.9e-btf/etc
+./bbchkcfg.sh
+(y to questions)
+./bbchkhosts.sh
+(ignore ssh errors)
+cd ../..
+chown -R bb .
+su bb
+cd
+cd bbc1.9e-btf/src
+make; make install
+cd ..
+vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
+        $1 $TOPARGS > $BBTMP/TOP.$$
+#        /usr/local/jail/bin/jtop > $BBTMP/TOP.$$
+./runbb.sh start
+more BBOUT
+(look for errors)
+exit
+echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
+chmod +x /usr/local/etc/rc.d/bb.sh
+Punch a hole in the firewall to allow it to communicate with bb monitor:
+ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2
+. configure bb on mail:
+vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
+.181.18.3 firewall2.johncompanies.com # ssh
+su bb
+cd
+bbsrc/bb/runbb.sh restart ; exit
+. configure ntp
+echo "server 10.1.2.1" > /etc/ntp.conf
+/usr/sbin/ntpd -p /var/run/ntpd.pid
+sleep 2; ntpq -p
+(confirm it’s able to reach our time server)
+echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
+chmod 0700 /usr/local/etc/rc.d/ntp.sh
+. fwd and reverse lookups on ns1c
+vr johncompanies.com
+ (edit the PTR too)
+. setup backups
+echo '#\!/bin/sh\
+backupdir=/data/firewall2/current\
+\
+## ENTRY /etc ' > /usr/local/etc/backup.config
+on backup3:
+setup backup dirs:
+ssh backup3 mkdir -p /data/firewall2/current
+on backup3, add the system to
+vi /usr/local/sbin/snapshot_archive
+scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
+vi /usr/local/etc/rsync.backup
+backup1 > backup3
+crontab -e
+0 * * * /usr/local/etc/rsync.backup
+. mkdir /root/logs
+. edit sshd_config for security
+vi /etc/ssh/sshd_config
+ListenAddress 66.181.18.3
+ListenAddress 10.1.2.1
+kill -1 `cat /var/run/sshd.pid`
+. raid chk
+cat > /usr/local/sbin/lsiraidchk
+#!/usr/bin/perl
+my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;
+foreach (@out) {
+    if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
+#print $_;
+}
+. add crontab entries
+crontab -e
+3 * * * /usr/local/etc/rsync.backup
+0 1 * * /sbin/ipfw zero
+0 1 * * /sbin/ipfw del 3 4 5
+23 30 * * /sbin/ipfw show > /tmp/ipfw_count
+0 30 * * /sbin/ipfw show > /tmp/ipfw_count
+3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
+*/5 * * * * /usr/local/sbin/lsiraidchk
+#10 0 * * * rm /var/spool/clientmqueue/*
+scp /etc/makefwrules.pl user@64.163.14.48:~
+scp /etc/makepiperules.pl user@64.163.14.48:~
+mv /home/user/makefwrules.pl /etc
+mv /home/user/makepiperules.pl /etc
+touch /etc/firewall.sh
+mkdir /etc/oldrules/
+other binaries
+scp /usr/local/bin/rulemaker user@64.163.14.48:~
+mv ~user/rulemaker /usr/local/sbin
+scp ~user/Sendmail.pm user@64.163.14.48:~
+scp ~user/doswatch.pl user@64.163.14.48:~
+Setup basic ruleset
+ipfw add 00009 count udp from any to any
+ipfw add 00010 allow tcp from any to any established
+ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
+ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
+ipfw add 00012 deny tcp from any to any tcpflags syn,fin
+ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
+ipfw add 00012 allow icmp from any to any
+ipfw add 00014 deny tcp from any to any dst-port 135
+ipfw add 00150 skipto 65535 ip from any to any via em1 in
+IPKVM3:
+allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
+deny ip from any to 69.55.230.10 dst-port 139
+</pre>
+= firewall3 =
+== Summary ==
+This machine is the backup firewall for the network at i2b.
+* Location: i2b, cab ?
+* OS: FreeBSD 9.1 amd64
+* Networking: Priv IP: 10.1.2.5, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks.
+<pre>
+The internal network NIC is the left one on the motherboard (69.55.229.1/24, ...).
+The external network NIC is the right one on the motherboard (66.181.18.3/28).
+The PCI ethernet card is connected to our private network (10.1.2.5/24).
+</pre>
+* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
+* Drives: 160 GB (2 x 160GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.
+== Services Provided ==
+* firewall (ipfw)
+* bigbrother
+== Firewall Rule Configuration ==
+See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.
+== Disaster Recovery ==
+'''To put the backup firewall3 into service:'''
+<pre>
+Move the internal cable (to our networks) from firewall2 to em1 which is the left most ethernet port (69.55.229.1).
+Move the external cable (to outside world) from firewall2 to em0 which is the port to the right on the motherboard (66.181.18.3).
+The PCI ethernet port (fxp0) should already be connected to private network (10.1.2.5).
+</pre>
+Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)
+<pre>
+kern_securelevel_enable="NO"
+portmap_enable="NO"
+sendmail_enable="NO"
+usbd_enable="YES"
+gateway_enable="YES"
+xntpd_enable="YES"
+nfs_client_enable="YES"
+nfs_reserved_port_only="YES"
+inetd_enable="YES"
+inetd_flags="-wW -a 10.1.2.5"
+fsck_y_enable="YES"
+background_fsck="NO"
+defaultrouter="66.181.18.2"
+hostname="firewall3.johncompanies.com"
+ifconfig_em0="inet 66.181.18.3  netmask 255.255.255.224"
+ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
+ifconfig_em1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
+# ifconfig_em1_alias1="inet 65.50.228.1  netmask 255.255.255.0"
+# ifconfig_em1_alias2="inet 65.50.229.1  netmask 255.255.255.0"
+# ifconfig_em1_alias3="inet 65.50.230.1  netmask 255.255.255.0"
+# ifconfig_em1_alias4="inet 65.50.231.1  netmask 255.255.255.0"
+# ifconfig_em1_alias5="inet 65.50.232.1  netmask 255.255.255.0"
+# ifconfig_em1_alias6="inet 65.50.233.1  netmask 255.255.255.0"
+# ifconfig_em1_alias7="inet 65.50.234.1  netmask 255.255.255.0"
+# ifconfig_em1_alias8="inet 65.50.235.1  netmask 255.255.255.0"
+ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"
+sshd_enable="YES"
+usbd_enable="YES"
+</pre>
+== Cronjobs ==
+3 * * * /usr/local/etc/rsync.backup
+Backup to backup3
+0 1 * * /sbin/ipfw zero
+0 1 * * /sbin/ipfw del 3 4 5
+Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).
+23 30 * * /sbin/ipfw show > /tmp/ipfw_count
+0 30 * * /sbin/ipfw show > /tmp/ipfw_count
+Capture counts periodically
+3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
+This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.
+ */5 * * * * /usr/local/sbin/lsiraidchk
+Checking the health of the RAID array
+== DOS attacks ==
+See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.
+== build ==
+<pre>partition map:
+/ 58g
+swap 4g
+/var 512m
+/tmp 512m
+/usr 5.5g
+. edit /etc/make.conf
+echo "WITHOUT_X11=yes \
+KERNCONF=firewall3 \
+BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf
+. add settings to /boot/loader.conf and /boot.config
+echo "-Dh" >> /boot.config
+echo 'console="comconsole,vidconsole" \
+boot_multicons="YES" \
+boot_serial="YES" \
+comconsole_speed="115200"' >> /boot/loader.conf
+. turn off all ttyv's except 0 and 1 in /etc/ttys
+also turn on ttyd0, change type to vt100:
+vi /etc/ttys
+ttyv2   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv3   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv4   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv5   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv6   "/usr/libexec/getty Pc"         cons25  off secure
+ttyv7   "/usr/libexec/getty Pc"         cons25  off secure
+# Serial terminals
+# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
+ttyd0   "/usr/libexec/getty std.9600"   vt100   on  secure
+kill -1 1
+on console server:
+vi /etc/remote
+(rename port to jail8 depending on where and which digi plugged into)
+test serial console
+. populate hosts
+echo "69.55.230.10 backup2" >> /etc/hosts
+echo "69.55.230.11 backup1" >> /etc/hosts
+echo "10.1.2.3 backup3" >> /etc/hosts
+. put key in authorized_keys on backup3
+cd
+ssh-keygen -t dsa -b 1024
+(default location, leave password blank)
+Punch a hole in firewall1 to allow traffic to backup servers @ castle:
+ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
+ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22
+cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
+cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
+cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
+confirm that you can ssh to backup3 and backup 2 without getting a login prompt
+ssh backup3 hostname
+ssh backup2 hostname
+ssh backup1 hostname
+. edit root's path and login script:
+vi /root/.cshrc
+Change alias entries (add G):
+alias la        ls -aG
+alias lf        ls -FAG
+alias ll        ls -lAG
+alias ls        ls -AG
+alias mbm       mb mount
+alias mbu       mb umount
+and alter the prompt, set the following:
+set prompt = "`/bin/hostname -s` %/# "
+. install cvsup
+cd /usr/ports/net/cvsup-without-gui
+make install clean; rehash; mail -s 'cvs installed' 8583619553@vtext.com < /dev/null
+. get latest sources for this release:
+cd /usr/src
+echo "*default host=cvsup4.freebsd.org\
+*default base=/usr\
+*default prefix=/usr\
+*default release=cvs tag=RELENG_9_1\
+*default delete use-rel-suffix\
+*default compress\
+src-all" > sup
+cvsup sup ; mail -s 'cvs sup done' 8583619553@vtext.com < /dev/null
+. configure new kernel.
+cd /usr/src/sys/amd64/conf
+scp backup2:/mnt/data4/build/freebsd/firewall3-9.1 ./firewall3
+. build, install kernel and world
+cd /boot
+mv kernel kernel.GENERIC
+cd kernel.GENERIC
+cd /usr/src
+make buildkernel installkernel
+make buildworld ; mail -s 'buildworld done' 8583619553@vtext.com < /dev/null
+(supermicro: 2:15 mins, 2950: 38? mins)
+make installworld
+(2450: 3min, supermicro: 1min, 2950: :34)
+mergemaster -i
+. populate /etc/rc.conf with IPs and NFS settings
+vi /etc/rc.conf
+kern_securelevel_enable="NO"
+portmap_enable="NO"
+sendmail_enable="NO"
+usbd_enable="YES"
+gateway_enable="YES"
+xntpd_enable="YES"
+nfs_client_enable="YES"
+nfs_reserved_port_only="YES"
+fsck_y_enable="YES"
+background_fsck="NO"
+hostname="firewall3.johncompanies.com"
+#  external network
+ifconfig_em0="inet 66.181.18.3  netmask 255.255.255.224"
+#  internal network
+ifconfig_em1="inet 69.55.229.1 netmask 255.255.255.0"
+ifconfig_em1_alias0="inet 69.55.231.1  netmask 255.255.255.0"
+ifconfig_em1_alias1="inet 65.50.228.1  netmask 255.255.255.0"
+ifconfig_em1_alias2="inet 65.50.229.1  netmask 255.255.255.0"
+ifconfig_em1_alias3="inet 65.50.230.1  netmask 255.255.255.0"
+ifconfig_em1_alias4="inet 65.50.231.1  netmask 255.255.255.0"
+ifconfig_em1_alias5="inet 65.50.232.1  netmask 255.255.255.0"
+ifconfig_em1_alias6="inet 65.50.233.1  netmask 255.255.255.0"
+ifconfig_em1_alias7="inet 65.50.234.1  netmask 255.255.255.0"
+ifconfig_em1_alias8="inet 65.50.235.1  netmask 255.255.255.0"
+defaultrouter="66.181.18.2"
+#  private network
+ifconfig_fxp0="inet 10.1.2.5 netmask 255.255.255.0"
+inetd_enable="YES"
+inetd_flags="-wW -a 10.1.2.5"
+sshd_enable="YES"
+usbd_enable="YES"
+ntpd_enable="YES"
+# powerd_enable="YES"
+. reboot. Confirm new kernel is loaded
+uname -a
+. update ports:
+cd /usr/ports
+echo "*default host=cvsup4.FreeBSD.org\
+*default base=/usr\
+*default prefix=/usr\
+*default release=cvs tag=RELENG_9_1\
+*default delete use-rel-suffix\
+*default compress\
+ports-all tag=." > sup
+cvsup sup; mail -s 'cvs sup ports done' 8583619553@vtext.com < /dev/null
+. Install raid mgmt tool
+# linux base
+cd /usr/ports/devel/libtool22
+make install base
+cd /usr/ports/emulators/linux_base-fc4
+make install clean
+scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz /usr/local/sbin
+cd /usr/local/sbin
+tar xzvf tw_cli-freebsd-x86_64-9.5.0.1.tgz
+rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
+. install rsync from ports
+cd /usr/ports/net/rsync
+make install clean
+choose default options
+. install bb client
+adduser
+Username: bb
+Full name: bb
+Uid (Leave empty for default): 1984
+Login group [bb]:
+Login group is bb. Invite bb into other groups? []:
+Login class [default]:
+Shell (sh csh tcsh nologin) [sh]:
+Home directory [/home/bb]:
+Use password-based authentication? [yes]:
+Use an empty password? (yes/no) [no]:
+Use a random password? (yes/no) [no]: yes
+Lock out the account after creation? [no]:
+Username   : bb
+Password   : <random>
+Full Name  : bb
+Uid        : 1984
+Class      :
+Groups     : bb
+Home       : /home/bb
+Shell      : /bin/sh
+Locked     : no
+OK? (yes/no): yes
+cd /usr/home/bb
+scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
+tar xvf bb-freebsd.tar
+edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
+echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
+.1.2.5 firewall3.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
+vi /home/bb/bbc1.9e-btf/ext/openfiles
+MACHINE="firewall3,johncompanies,com"      # HAS TO BE IN A,B,C FORM
+cd /usr/home/bb/bbc1.9e-btf/etc
+./bbchkcfg.sh
+(y to questions)
+./bbchkhosts.sh
+(ignore ssh errors)
+cd ../..
+chown -R bb .
+su bb
+cd
+cd bbc1.9e-btf/src
+make; make install
+cd ..
+vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
+        $1 $TOPARGS > $BBTMP/TOP.$$
+#        /usr/local/jail/bin/jtop > $BBTMP/TOP.$$
+./runbb.sh start
+more BBOUT
+(look for errors)
+exit
+echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
+chmod +x /usr/local/etc/rc.d/bb.sh
+Punch a hole in the firewall to allow it to communicate with bb monitor:
+ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2
+. configure bb on mail:
+vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
+.181.18.3 firewall3.johncompanies.com # ssh
+su bb
+cd
+bbsrc/bb/runbb.sh restart ; exit
+. configure ntp
+echo "server 10.1.2.1" > /etc/ntp.conf
+/usr/sbin/ntpd -p /var/run/ntpd.pid
+sleep 2; ntpq -p
+(confirm it’s able to reach our time server)
+echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
+chmod 0700 /usr/local/etc/rc.d/ntp.sh
+. fwd and reverse lookups on ns1c
+vr johncompanies.com
+ (edit the PTR too)
+. setup backups
+echo '#\!/bin/sh\
+backupdir=/data/firewall2/current\
+\
+## ENTRY /etc ' > /usr/local/etc/backup.config
+on backup3:
+setup backup dirs:
+ssh backup3 mkdir -p /data/firewall2/current
+on backup3, add the system to
+vi /usr/local/sbin/snapshot_archive
+scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
+vi /usr/local/etc/rsync.backup
+backup1 > backup3
+crontab -e
+0 * * * /usr/local/etc/rsync.backup
+. mkdir /root/logs
+. edit sshd_config for security
+vi /etc/ssh/sshd_config
+ListenAddress 66.181.18.3
+ListenAddress 10.1.2.5
+kill -1 `cat /var/run/sshd.pid`
+. raid chk
+cat > /usr/local/sbin/lsiraidchk
+#!/usr/bin/perl
+my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;
+foreach (@out) {
+    if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
+#print $_;
+}
+. add crontab entries
+crontab -e
+3 * * * /usr/local/etc/rsync.backup
+0 1 * * /sbin/ipfw zero
+0 1 * * /sbin/ipfw del 3 4 5
+23 30 * * /sbin/ipfw show > /tmp/ipfw_count
+0 30 * * /sbin/ipfw show > /tmp/ipfw_count
+3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
+*/5 * * * * /usr/local/sbin/lsiraidchk
+#10 0 * * * rm /var/spool/clientmqueue/*
+scp /etc/makefwrules.pl user@64.163.14.48:~
+scp /etc/makepiperules.pl user@64.163.14.48:~
+mv /home/user/makefwrules.pl /etc
+mv /home/user/makepiperules.pl /etc
+touch /etc/firewall.sh
+mkdir /etc/oldrules/
+other binaries
+scp /usr/local/bin/rulemaker user@64.163.14.48:~
+mv ~user/rulemaker /usr/local/sbin
+scp ~user/Sendmail.pm user@64.163.14.48:~
+scp ~user/doswatch.pl user@64.163.14.48:~
+Setup basic ruleset
+ipfw add 00009 count udp from any to any
+ipfw add 00010 allow tcp from any to any established
+ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
+ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
+ipfw add 00012 deny tcp from any to any tcpflags syn,fin
+ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
+ipfw add 00012 allow icmp from any to any
+ipfw add 00014 deny tcp from any to any dst-port 135
+ipfw add 00150 skipto 65535 ip from any to any via em1 in
+IPKVM3:
+allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
+deny ip from any to 69.55.230.10 dst-port 139
+</pre>
+= wiki =
+The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8
+The backup wiki lives on virt13 in CT 5 / 69.55.230.18
+== Setup jail ==
+<pre>
+mkdir /mnt/data1/wiki-dir
+cd /usr/src
+make installworld DESTDIR=/mnt/data1/wiki-dir
+cd etc
+make distribution DESTDIR=/mnt/data1/wiki-dir
+mount -t devfs devfs /mnt/data1/wiki-dir/dev
+devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
+cd /mnt/data1/wiki-dir
+ln -sf dev/null kernel
+scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin
+jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
+csh
+touch /etc/fstab
+echo 'network_interfaces=""\
+hostname="wiki.johncompanies.com"\
+kern_securelevel_enable="NO"\
+sendmail_enable="YES"\
+sshd_enable="YES"' > /etc/rc.conf
+echo "nameserver 69.55.229.3\
+nameserver 69.55.225.225" >> /etc/resolv.conf
+vi /etc/crontab
+(remove the adjkerntz lines )
+vi /etc/periodic/security/100.chksetuid
+replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
+ with: MP='/' (use single quotes)
+mkdir -p /usr/compat/linux/dev
+adduser
+Username: user
+Full name: user
+Uid (Leave empty for default):
+Login group [user]:
+Login group is user. Invite user into other groups? []: wheel
+Login class [default]:
+Shell (sh csh tcsh nologin) [sh]:
+Home directory [/home/user]:
 Home directory permissions (Leave empty for default):
 Use password-based authentication? [yes]:
 Use an empty password? (yes/no) [no]:
 Use a random password? (yes/no) [no]: y
 Lock out the account after creation? [no]:
 Username   : user
 Password   : <random>
 Full Name  : user
 Uid        : 1001
 Class      :
 Groups     : user
 Home       : /home/user
 Home Mode  :
 Shell      : /bin/sh
 Locked     : no
 OK? (yes/no): y
 adduser: INFO: Successfully added (user) to the user database.
 adduser: INFO: Password for (user) is: 901gmYjO
 Add another user? (yes/no): n
 Goodbye!
+vi /usr/home/user/.profile
+TERM=vt100;     export TERM
+tzsetup
+newaliases
+rm /sbin/halt /sbin/reboot
+ln /sbin/jkill /sbin/halt
+ln /sbin/jkill /sbin/reboot
+vi /etc/syslog.conf
+#*.err;kern.warning;auth.notice;mail.crit               /dev/console
+*.err;kern.warning;auth.notice;mail.crit                /var/log/messages
+exit
+exit
+cd libexec
+chflags noschg ld-elf32.so.1
+chflags noschg ld-elf.so.1
+mv ld-elf32.so.1 ld-elf32.so.1-orig
+ln ld-elf.so.1 ld-elf32.so.1
+chflags schg ld-elf.so.1
+chflags schg ld-elf32.so.1
+cp -r /usr/ports /mnt/data1/wiki-dir/usr
+cat > /usr/local/etc/rc.d/wiki.sh
+mount -t devfs devfs /mnt/data1/wiki-dir/dev/
+devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
+jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc
+chmod 0700 /usr/local/etc/rc.d/wiki.sh
+</pre>
+== mediawiki setup ==
+<pre>
+cd /usr/ports/net/rsync
+make install clean
+cd /usr/ports/distfiles/
+fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
+cd /usr/ports/databases/mysql55-server
+make install clean
+cd /usr/ports/distfiles/
+fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
+cd /usr/ports/lang/php52
+make install clean
+(build apache module)
+cd /usr/ports/lang/php5-extensions
+make install clean
+cd /usr/ports/www/apache22
+make install clean
+cd /usr/local/www/
+fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
+tar xzf mediawiki-1.19.2.tar.gz
+mv mediawiki-1.19.2 wiki
+vi /usr/local/etc/apache22/httpd.conf
+DocumentRoot "/usr/local/www/"
+Include etc/apache22/extra/vhost-wiki.conf
+Listen 443
+<IfModule mod_php5.c>
+    <FilesMatch "\.ph(p3?|tml)$">
+        SetHandler application/x-httpd-php
+    </FilesMatch>
+    <FilesMatch "\.phps$">
+        SetHandler application/x-httpd-php-source
+    </FilesMatch>
+    # To re-enable php in user directories comment the following lines
+    # (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
+    # prevents .htaccess files from disabling it.
+    <IfModule mod_userdir.c>
+        <Directory /home/*/public_html>
+            php_admin_value engine Off
+        </Directory>
+    </IfModule>
+</IfModule>
+cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
+<VirtualHost *:443>
+        ServerAdmin support@johncompanies.com
+        DocumentRoot /usr/local/www/wiki
+#        <Directory />
+#                Options FollowSymLinks
+#                AllowOverride None
+#                Order deny,allow
+#        </Directory>
+        <Directory /usr/local/www/wiki>
+                Options Indexes FollowSymLinks MultiViews
+                Deny from all
+                AllowOverride AuthConfig
+                Order allow,deny
+                DirectoryIndex index.php
+                #Allow from 69.55.233.195
+                #Allow from boody.dyndns.org
+        </Directory>
+        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
+        <Directory "/usr/lib/cgi-bin">
+                AllowOverride None
+                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
+                Order allow,deny
+                Allow from all
+        </Directory>
+        ErrorLog /var/log/httpd-error.log
+        # Possible values include: debug, info, notice, warn, error, crit,
+        # alert, emerg.
+        LogLevel warn
+        CustomLog /var/log/httpd-access.log combined
+    Alias /doc/ "/usr/share/doc/"
+    <Directory "/usr/share/doc/">
+        Options Indexes MultiViews FollowSymLinks
+        AllowOverride None
+        Order deny,allow
+        Deny from all
+        Allow from 127.0.0.0/255.0.0.0 ::1/128
+    </Directory>
+    SSLEngine on
+    SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
+    SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key
+</VirtualHost>
+mkdir ssl
+cd ssl
+openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
+US
+CA
+San Diego
+johncompanies.com
+johncompanies.com
+wiki.johncompanies.com
+support@johncompanies.com
+cat > /usr/local/www/wiki/.htaccess
+AuthType Basic
+AuthUserFile /usr/local/etc/apache22/wiki.passwd
+AuthName wiki
+require valid-user
+satisfy any
+cd /usr/local/etc/apache22
+htpasswd -c wiki.passwd admin
+https://69.55.229.8/index.php
+use mysql (innodb)
+wiki name: JCWiki
+Support / (mail pass) / support@johncompanies.com
+cat > /usr/local/www/wiki/LocalSettings.php
+<?php
+# This file was automatically generated by the MediaWiki 1.19.2
+# installer. If you make manual changes, please keep track in case you
+# need to recreate them later.
+#
+# See includes/DefaultSettings.php for all configurable settings
+# and their default values, but don't forget to make changes in _this_
+# file, not there.
+#
+# Further documentation for configuration settings may be found at:
+# http://www.mediawiki.org/wiki/Manual:Configuration_settings
+# Protect against web entry
+if ( !defined( 'MEDIAWIKI' ) ) {
+	exit;
+}
+## Uncomment this to disable output compression
+# $wgDisableOutputCompression = true;
+$wgSitename      = "JCWiki";
+## The URL base path to the directory containing the wiki;
+## defaults for all runtime URL paths are based off of this.
+## For more information on customizing the URLs please see:
+## http://www.mediawiki.org/wiki/Manual:Short_URL
+$wgScriptPath       = "";
+$wgScriptExtension  = ".php";
+## The protocol and server name to use in fully-qualified URLs
+$wgServer           = "https://69.55.229.8";
+## The relative URL path to the skins directory
+$wgStylePath        = "$wgScriptPath/skins";
+## The relative URL path to the logo.  Make sure you change this from the default,
+## or else you'll overwrite your logo when you upgrade!
+#$wgLogo             = "$wgStylePath/common/images/wiki.png";
+$wgLogo             = "$wgStylePath/common/images/jclogo.gif";
+## UPO means: this is also a user preference option
+$wgEnableEmail      = true;
+$wgEnableUserEmail  = true; # UPO
+$wgEmergencyContact = "apache@69.55.229.8";
+$wgPasswordSender   = "apache@69.55.229.8";
+$wgEnotifUserTalk      = false; # UPO
+$wgEnotifWatchlist     = false; # UPO
+$wgEmailAuthentication = true;
+## Database settings
+$wgDBtype           = "mysql";
+$wgDBserver         = "localhost";
+$wgDBname           = "my_wiki";
+$wgDBuser           = "root";
+$wgDBpassword       = "";
+# MySQL specific settings
+$wgDBprefix         = "";
+# MySQL table options to use during installation or update
+$wgDBTableOptions   = "ENGINE=InnoDB, DEFAULT CHARSET=binary";
+# Experimental charset support for MySQL 5.0.
+$wgDBmysql5 = false;
+## Shared memory settings
+$wgMainCacheType    = CACHE_NONE;
+$wgMemCachedServers = array();
+## To enable image uploads, make sure the 'images' directory
+## is writable, then set this to true:
+$wgEnableUploads  = false;
+#$wgUseImageMagick = true;
+#$wgImageMagickConvertCommand = "/usr/bin/convert";
+# InstantCommons allows wiki to use images from http://commons.wikimedia.org
+$wgUseInstantCommons  = false;
+## If you use ImageMagick (or any other shell command) on a
+## Linux server, this will need to be set to the name of an
+## available UTF-8 locale
+$wgShellLocale = "en_US.utf8";
+## If you want to use image uploads under safe mode,
+## create the directories images/archive, images/thumb and
+## images/temp, and make them all writable. Then uncomment
+## this, if it's not already uncommented:
+#$wgHashedUploadDirectory = false;
+## Set $wgCacheDirectory to a writable directory on the web server
+## to make your wiki go slightly faster. The directory should not
+## be publically accessible from the web.
+#$wgCacheDirectory = "$IP/cache";
+# Site language code, should be one of the list in ./languages/Names.php
+$wgLanguageCode = "en";
+$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";
+# Site upgrade key. Must be set to a string (default provided) to turn on the
+# web installer while LocalSettings.php is in place
+$wgUpgradeKey = "3196710f4a7d7332";
+## Default skin: you can change the default skin. Use the internal symbolic
+## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
+$wgDefaultSkin = "vector";
+## For attaching licensing metadata to pages, and displaying an
+## appropriate copyright notice / icon. GNU Free Documentation
+## License and Creative Commons licenses are supported so far.
+$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
+$wgRightsUrl  = "";
+$wgRightsText = "";
+$wgRightsIcon = "";
+# Path to the GNU diff3 utility. Used for conflict resolution.
+$wgDiff3 = "/usr/bin/diff3";
+# Query string length limit for ResourceLoader. You should only set this if
+# your web server has a query string length limit (then set it to that limit),
+# or if you have suhosin.get.max_value_length set in php.ini (then set it to
+# that value)
+$wgResourceLoaderMaxQueryLength = -1;
+# End of automatically generated settings.
+# Add more configuration options below.
+</pre>
+== copy/backup wiki ==
+on main/primary wiki:
+<pre>
+/usr/local/etc/rc.d/mysql-server stop
+ssh 69.55.230.18 "/etc/init.d/mysql stop"
+rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
+rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/
+/usr/local/etc/rc.d/mysql-server start
+ssh 69.55.230.18 "/etc/init.d/mysql start"
-vi /usr/home/user/.profile
-TERM=vt100;     export TERM
-tzsetup
-newaliases
-rm /sbin/halt /sbin/reboot
-ln /sbin/jkill /sbin/halt
-ln /sbin/jkill /sbin/reboot
-vi /etc/syslog.conf
-#*.err;kern.warning;auth.notice;mail.crit               /dev/console
-*.err;kern.warning;auth.notice;mail.crit                /var/log/messages
-exit
-exit
-cd libexec
-chflags noschg ld-elf32.so.1
-chflags noschg ld-elf.so.1
-mv ld-elf32.so.1 ld-elf32.so.1-orig
-ln ld-elf.so.1 ld-elf32.so.1
-chflags schg ld-elf.so.1
-chflags schg ld-elf32.so.1
-cp -r /usr/ports /mnt/data1/wiki/usr
 </pre>

Infrastructure Machines: Difference between revisions

Latest revision as of 15:05, 3 October 2023

jails[edit]

jail1[edit]

jail2[edit]

jail3[edit]

Notes[edit]

jail4[edit]

Notes[edit]

jail5[edit]

Notes[edit]

jail6[edit]

jail7[edit]

Notes[edit]

jail8[edit]

jail9[edit]

jail11[edit]

mx1[edit]

Notes[edit]

mx2[edit]

Notes[edit]

jail17[edit]

jail18[edit]

jail19[edit]

virts[edit]

quar1[edit]

Notes[edit]

virt9[edit]

virt11[edit]

virt12[edit]

virt13[edit]

Notes[edit]

virt14[edit]

Notes[edit]

virt15[edit]

virt16[edit]

virt17[edit]

virt19[edit]

mail[edit]

Summary[edit]

Services Provided[edit]

email[edit]

IP Blocking[edit]

web[edit]

mysql[edit]

bigbrother[edit]

DNS (ns1c.johncompanies.com)[edit]

Usage and Notes[edit]

Cronjobs[edit]

Regular maintenance[edit]

Building a new Mail Server[edit]

Installations[edit]

Web Server[edit]

Mail Server[edit]

DNS Server (ns1c.johncompanies.com)[edit]

ns2c[edit]

Summary[edit]

nat[edit]

Summary[edit]

Services Provided[edit]

nat control[edit]

nat2[edit]

Summary[edit]

Services Provided[edit]

nat config[edit]

build[edit]

bwdb[edit]

Summary[edit]

Services Provided[edit]

netflow[edit]

mysql[edit]

Regular maintenance[edit]

Slaving[edit]

Build[edit]

BIOS Config[edit]

Install OS[edit]

netflow[edit]

process flow tools[edit]

flow processing: i2b[edit]

flow processing: castle[edit]