JCWiki - User contributions [en]

Uncancel in MGMT

2013-04-02T21:28:36Z

76.88.21.129:

this is a manual database update to the following tables: fields

* customers: status->1
* ipmap: (for each ip) relen_date->null, status->1
* subs: (for last/current sub(s)) status->1, stop_date->null
* systems: status->1, stop_date->null, cancel_date->null

Uncancel in MGMT

2013-04-02T21:27:58Z

76.88.21.129:

this is a manual database update to the following tables: fields

* customers: status->1
* ipmap: (for each ip) relen_date->null, status->1
* subs: status->1, stop_date->null
* systems: status->1, stop_date->null, cancel_date->null

Payment System

2013-03-30T15:52:16Z

76.88.21.129: /* Cancelled PayPal Subs */

= Overview / Billing System =

We handle payments in a fairly straightforward manner- each customer has a running ledger ([[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:billing_history|jc:billing_history]]). Any negative balance in the ledger will trigger the customer to be billed that amount to whatever payment source is on file, provided one exists. A positive balance will remain on file as a credit towards future service charges.

The [[Infrastructure_Machines#Cronjobs|biller.pl]] script will look through the [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:subs|jc:subs]] database at 00:25 PT to gather up service charges due as of the time/date the biller.pl script is run. Those amounts will add to a negative (amount due) balance.

The [[Infrastructure_Machines#Cronjobs|pfp_batch_gather.pl]] script will run at 13:10 PT to look for customers with a balance due and an active ([[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:pmt_source|jc:pmt_source.status=1]] and [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:pmt_source|jc:pmt_source.type='f']]) credit card pmt source, and build a list of items ([[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc.pfp_batch|jc.pfp_batch]]) to charge via payflow.

The [[Infrastructure_Machines#Cronjobs|pfp_batch_process.pl]] script will run at 14:10 PT to process whatever is in [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc.pfp_batch|jc.pfp_batch]].

The [[Infrastructure_Machines#Cronjobs|pb_batch_gather.pl]] script will run at 13:15 PT to look for customers with a balance due and an active ([[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:pmt_source|jc:pmt_source.status=1]] and [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc:pmt_source|jc:pmt_source.type='b']]) paypal (billing agreement) pmt source, and build a list of items ([[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc.pb_batch|jc.pb_batch]]) to charge paypal.

The [[Infrastructure_Machines#Cronjobs|pb_batch_process.pl]] script will run at 14:15 PT to process whatever is in [[Management_System_/_Public_Website_/_Signup_/_Account_Manager#jc.pb_batch|jc.pb_batch]].

= PayFlow =
== Performing a refund ==
Refunding money via PayFlow must be done manually (simply adding a credit to a ledger will not result in funds going back on a card, it will simply carry forward as a credit towards future charges). To perform a refund, first login: https://manager.paypal.com/

Go to the virtual terminal, choose single transaction, transaction type: Credit

In the original transaction ID field enter the payment ref from the customer's billing information, e.g. ELEP7DF85E73
Credit Card Number should be left blank. After submitting enter the amount to refund, all other fields should be left as is. If the amount is too big (i.e. above the amount we have set for refunds, you'll need to raise that limit in the account settings, then retry the refund).

After the refund is processed, you should enter it as a payment (negative amount) and copy over the transaction ID from the refund, into the customer's payments so we have a record of it. To do this: management -> (customer's page) edit/view sources/pmts -> add payment -> enter amount, date, pnref (unique transaction ID of the refund transaction), result code (0 for success), vendor (PayFlow), leave add to ledger checked (so it will show up in billing history).

== Card Data ==

In the event we ever have to re-enter all credit card data (i.e. moving to a new card processor) we encrypt and store card data for each transaction coming in. Each time card info comes in it is stored in <tt>/usr/local/www/scripts/ccard_orders/done/xxxxxx.gpg</tt> If the data comes in as the result of a signup, the file name is randomized and must be renamed with the correct customer ID. If the data comes in from the card input page or via the AM, then it will be correctly named with the customer ID. To decrypt the data:
(as root):
mail /home/support# gpg --decrypt /usr/local/www/ccard_orders/done/xxxxxx.gpg

= Payment-related emails =
== "No Payment Received for JohnCompanies account - Action Required - WARNING - SHUTDOWN IMMINENT" ==

<pre>(NOTE: this is the 4th notice we have sent in 6 weeks - please respond immediately)

Re:

Hello,

Our records indicate that your billing is not current for your JohnCompanies account -
you may have recently changed billing information or perhaps initiated new service or
altered existing service.

This is not a problem - we simply need you to immediately visit your Account Manager via
this link:

https://secure.johncompanies.com/am/billing.html?cid=col02020

which contains your current and/or past-due charges in detail. Please do contact
payments@johncompanies.com if you have any questions at all.

Thank you so much for your continued business - we appreciate it greatly,

JohnCompanies Hosting
</pre>

We send out automatic notices to customers who were paying via credit card (have a negative ledger balance) and are marked in the database as in arrears. The send schedule is (subject addendum):
week 1 
week 3 ("SECOND NOTICE") 
week 5 ("THIRD NOTICE - PLEASE RESPOND") 
week 6 ("WARNING - SHUTDOWN IMMINENT") 
and weekly thereafter

'''Action to take:''' Once we see the WARNING - SHUTDOWN IMMINENT notice, we take action to put their server on shutdown in mgmt. We set a cancel date for 10 days in advance and set a note like: "20121024 - no pmt, setup shutdown"

== "batch gather report - 5 ($-1720)" ==
<pre>
accounts with balances: 50
items added to pfp batch queue: 5 ($-1720)
accounts with alternate (non-ccard) pmt source: 41 ($-16920.06)
</pre>

Shows you how many accounts have balances (50) how many items were added to the batch and their cumulative value (5, $1720) and the number of accounts and the amount due for customers with no valid payment source (41, $16920.06)

'''Action to take:''' Watch that the [[#.22batch_process_report_-_4_.28.241596.29.22|batch process report - 4 ($1596)]] email roughly matches what was gathered today. Delete email.

== "batch process report - 4 ($1596)" ==
<pre>
items processed: 9
items billed successfully: 4 ($1596)
items failed to bill: 5 ($629)
</pre>

Shows you how many items were in the queue. How many were billed successfully and the total amount (4, $1596), and the items that failed and their value (5, $629)

'''Action to take:'''Make sure ''something'' was processed today- maybe a payflow error? Delete email.

== "failed pfp_batch item for col01788" ==
<pre>
RESPMSG:Declined: 15005-This transaction cannot be
processed.;X-VPS-REQUEST-ID:1349989805Scol01788;CVV2MATCH:X;AVSADDR:N;PROCCVV2:I;PROCAVS:
N;HOSTCODE:15005;PNREF:EQFP7B38D1B3;RESULT:12;AVSZIP:N;IAVS:N;
</pre>

Indicates we have tried to bill this card 5 times and it has failed each time, and now the customer has been placed in arrears (will get automatic email reminders with payment link) and their charge has been removed form the payflow batch queue [[https://secure.johncompanies.com/mgmt/pfp_batch.html https://secure.johncompanies.com/mgmt/pfp_batch.html]]
In the body of the email you can see details about the reason for failure, usually a decline (as in this example).

'''Action to take:'''Make sure it's a standard decline and nothing else wrong. Customer is in arrears, they'll be emailed. Delete email.

== "Subscription cancelled for S-1111111111111" ==
'''Action to take:''' See [[Payment_System#Cancelled_PayPal_Subs|Cancelled PayPal Subs]]. Delete email.

== "successful pmt for col01111" ==

<pre>
RESPMSG:Approved;PPREF:0RH20660AL5779445;AUTHCODE:111111;X-VPS-REQUEST-ID:1349524679Scol0
2154;CVV2MATCH:Y;AVSADDR:Y;PROCCVV2:M;PROCAVS:Y;CORRELATIONID:20f200c78b60e;PNREF:ESYP9EC
9E9FF;RESULT:0;AVSZIP:Y;IAVS:N;
</pre>

This shows that we have a successful payment ('pmt') from this customer. You may search for the payment in PayFlow via the PNREF ID ('ESYP9EC9E9FF'). You can also see some fraud filter results in this email:

<tt>CVV2MATCH:Y</tt> cvv matches (Y=match, N=no match, X=not supported by bank)

<tt>AVSADDR:Y</tt> address verification matches (Y=match, N=zip match, X=not supported)

<tt>PROCCVV2:M</tt> verbose cvv match info (S=not supported, U=unavailable, W=whole zip, X=exact match, Y=yes, Z=zip)

<tt>PROCAVS:Y</tt> verbose address verification info (M=match, N=no match, P=not processed, S=not supported, U=unavailable, X=no response)

'''Action to take''': Clear arrears for customer. Delete email.

== "successful S for 68479f46" ==
This means a new customer paid via cc. '''S''' means we collected $. '''A''' means it's only an auth (in the case of a colo) and we will need to collect funds later.

'''Action to take''': rename the cc info file (see [[#.22New_ccard_order_68479f46.22|New ccard order 68479f46]]). Delete email

== "New ccard order 68479f46" ==

<pre>
From: Chad Whitacre
gpg --decrypt /usr/local/www/ccard_orders/done/43b7214f

-----BEGIN PGP MESSAGE-----
Version: Crypt::OpenPGP 1.03

hQEOA/J4/DszoRXaEAQAiNS5gAlrM0XQLMvUnJvq8GWQUptOZvXbIInoCYlWUrMG
ZMLoIuNLua6IlBjV2yJg+r0f8jhsMfsiD+kIZq7xo8sEVfSXaKtGNsi1K8KejUYh
ZKclJh4DULOeTRiumg0rsgbvldgs7JlHa/WfyawJjhD4oV8YJy5c6KyDic4hFmoD
/jmsi60I8bfkgMQ+5SRM8FcMKL4evuheRLvvFbI4f8SoOzR+LsQPsn+hfTsC/Qp8
X2YjCp6w2Oyxz5TZE4SMeVOP70aR1RK7dhr2NQPTr6pFNM3b3zGIgSx8AEs8pgsy
Yv+cVEjV/iUv+lFpBxqqoid53YNTx9pP+kPMjLLKJk6b0sAdAbxG68FtgnBybMui
gMbf92ji+IR87q0D16/nPjzsAv3s7zRw/nzZIOlgQ1/GAFEF7VSyQHzOkd9UQuS0
kU0XXrUpVi0v0PGc9nMWjBA5d2YsaqDrZRxFgzCqwBmhxMHAQqQMzfaEyQIL4EF0
LUpVe4cWiI3aSjPx0aC7bFrioqqXl8vJNfiFMgt1gY6J1PQ4S/wRgmLDNcJrejAA
483biJUsDF4S02JK3QLbUcYN/2NJE2M0/cO88DXXZvm6B7jc0URMN5ytYrv3SYHu
VGuNRCbdaicekD6VnF9EkUA=
=zKlm
-----END PGP MESSAGE-----
</pre>

This is the encrypted cc info. It's viewable in the email and placed in a file (<tt>/usr/local/www/ccard_orders/done/43b7214f</tt>) on mail. It needs to be renamed to the customer's final CID:

<tt>mv /usr/local/www/ccard_orders/done/43b7214f /usr/local/www/ccard_orders/done/col0NNNN.gpg</tt>

This file can be decrypted (gpg --decrypt /path/to/file) to get at the info.

'''Action to take''': rename the file. Delete email.

== "failed pmt for col01111" ==
<pre>
RESPMSG:Duplicate trans: 10536-The transaction was refused as a result of a duplicate
invoice ID supplied. Attempt with a new invoice
ID;PNREF:ETHPA078CA93;RESULT:30;X-VPS-REQUEST-ID:1349657543Scol01034;HOSTCODE:10536;
</pre>

Pretty self explanatory. Their payment failed, and they do know about it. This is sent to us so we know it happened, in case (in the case above) the fault was our own. Most of the time the reason will be a plain old DECLINE.

'''Action to take''': None, unless the fault is our own. The customer should go back to the payment page and try to pay again. Delete email.

== "Unknown IPN type:" ==

<pre>payer_email => lenderreviews@gmail.com
first_name => Randall
item_number => 0003
payment_gross => -39.00
mc_currency => USD
receiver_id => 8EKTVPV9MKWF4
ipn_track_id => 15729a938d26a
invoice => a9ca993f
subscr_id => S-5SK10643PA3743414
reason_code => other
business => payments@johncompanies.com
charset => windows-1252
mc_gross => -39.00
custom => type=pp
last_name => Marquis
notify_version => 3.7
payer_id => 9VGJXDXGDW2GC
residence_country => US
mc_fee => -0.74
payer_business_name => Lender Reviews, LLC
txn_id => 0N451285L6563193U
cmd => _notify-validate
verify_sign => AhWl7ACVsSRPc7TyALD6xAJDmOYcAWCn9nISbtkuorhGfhawdKvf6fsV
parent_txn_id => 4SC12038JA710930G
receiver_email => payments@johncompanies.com
item_name => JohnCompanies.com Colocation - $39.00/1 month(s)
payment_status => Reversed
protection_eligibility => Ineligible
payment_date => 12:02:18 Sep 26, 2012 PDT
payment_fee => -0.74
transaction_subject =>
payment_type => instant
</pre>

Paypal sends us notifications (IPN) for everything that happens with our Paypal account. Most of these IPNs are quietly handled by our billing system. Occasionally, something happens that our billing system doesn't recognize or our billing system wasn't made/meant to handle so it sends this email to let you decide what to do.

Looking through the parameters we see that this was a funds reversal: <tt>payment_status => Reversed</tt>

You can login to paypal and get more info about this by [https://history.paypal.com/us/cgi-bin/webscr?cmd=_history-search searching] on one of several items: 
payer email: <tt>lenderreviews@gmail.com</tt> 
transaction ID: <tt>0N451285L6563193U</tt> 
subscription ID: <tt>S-5SK10643PA3743414</tt> 

'''Action to take:''' customer needs to be re-invoiced and likely the sub was cancelled as well.

= Cancelled PayPal Subs =
When a customer cancels their sub we will get a notice about it:

<pre>
From: support@johncompanies.com
To: support@johncompanies.com
Subject: Subscription cancelled for S-9X6247331K616642Y

mc_amount1 => 189.00
mc_amount3 => 189.00
payer_email => kpsurplusdotcom@gmail.com
first_name => Ken
item_number => 0003
txn_type => subscr_cancel
mc_currency => USD
payer_status => verified
ipn_track_id => 48ead05d235f3
invoice => 71b20c79
reattempt => 1
subscr_id => S-9X6247331K616642Y
recurring => 1
business => payments@johncompanies.com
charset => windows-1252
custom => type=pp
last_name => Parris
notify_version => 3.7
period1 => 19 D
period3 => 1 M
payer_id => RMV4JADWCFYJW
residence_country => US
amount1 => 189.00
subscr_date => 03:08:14 Oct 09, 2012 PDT
payer_business_name => KP's Surplus
amount3 => 189.00
cmd => _notify-validate
verify_sign => Aoq5NOCTR258UimYDF1inxShI0akA-Ayb5I4ZayCGz8enxz3aUwfLv5c
receiver_email => payments@johncompanies.com
item_name => JohnCompanies.com Colocation - $189.00/1 month(s)
</pre>

A sub gets canceled for one of a few reasons:
# the customer took action to cancel it
# the customer's funding source(s) failed to provide funds to pay the subscription amount
# paypal sometimes cancels these for their own reasons
# the sub may have been setup (incorrectly) to make only a certain number of payments, then it auto-cancels

The following is all DEPRECATED. As of 20130301 we now simply refer people to the payments page and they pay whatever their balance due is.

Either way, if we take no action, then the customer will no longer be making payments for their service(s). So we need to take action to get them paying again. To get them paying again via subscription, we send them a payment reminder (invoice).

# search for the customer in management by the subID (<tt>S-9X6247331K616642Y</tt>) or payer's email (<tt>kpsurplusdotcom@gmail.com</tt>)
# take note of the last payment date for the sub and the nature of the sub payments: how much and how often.
# click the 're-invoice' link associated with the failed sub, this takes you to the Invoice Generator. It will be pre-filled with some ''assumptions'' based on the last payment date and their payment.
# confirm the payment, start date and interval. The Generator assumes monthly billing in all cases. So if they were paying annually, you will need to correct it. If the sub has been cancelled because paypal was unable to collect funds from their payment sources (most common reason) then the payment due date will have passed. In that case you will see a single amount due now (Single Amount), and the regular (monthly/annual) payment will show up in the 'Recurring Bill' field along with the next regular payment date- it will try to preserve their former monthly payment date. Again, if this is an annual payment it will guess wrong and the values will be incorrect so you must fix them and chose "bill every 12 months" too.
# in most cases we will allow the Invoice Generator to send out an invoice to the customer, so choose "yes, ..." in the 'Email invoice to customer?' field. The Generator will offer up some stock message formats. In the case of a sub cancelled with a missed payment, it will auto choose "last payment failed" and fill out the body appropriately. In the case of a sub cancelled, but no payment missed, you should make sure the message selected it "sub cxld". Depending on the message, it will prompt you to provide different fields (like "Amt of missed pmt") which it will insert into the message for you. Bottom line- you can write or use whatever message makes sense, just make sure the message includes the link to payment (which is provided just above).
# select the contacts you want to receive the message- usually just the billing contacts. You may also enter more (comma-delimited)
# in the 'Notifications' section you notice that support@ is auto-selected to receive notification when the bill is paid. This simply means when the customer pays, the corresponding address will be sent an email indicating the invoice was paid.

Infrastructure Machines

2013-03-25T02:42:48Z

76.88.21.129: /* mediawiki setup */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>

cd /usr/ports/net/rsync
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com

cat > /usr/local/www/wiki/LocalSettings.php

<?php
# This file was automatically generated by the MediaWiki 1.19.2
# installer. If you make manual changes, please keep track in case you
# need to recreate them later.
#
# See includes/DefaultSettings.php for all configurable settings
# and their default values, but don't forget to make changes in _this_
# file, not there.
#
# Further documentation for configuration settings may be found at:
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# Protect against web entry
if ( !defined( 'MEDIAWIKI' ) ) {
exit;
}

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

$wgSitename = "JCWiki";

## The URL base path to the directory containing the wiki;
## defaults for all runtime URL paths are based off of this.
## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
$wgScriptPath = "";
$wgScriptExtension = ".php";

## The protocol and server name to use in fully-qualified URLs
$wgServer = "https://69.55.229.8";

## The relative URL path to the skins directory
$wgStylePath = "$wgScriptPath/skins";

## The relative URL path to the logo. Make sure you change this from the default,
## or else you'll overwrite your logo when you upgrade!
#$wgLogo = "$wgStylePath/common/images/wiki.png";
$wgLogo = "$wgStylePath/common/images/jclogo.gif";

## UPO means: this is also a user preference option

$wgEnableEmail = true;
$wgEnableUserEmail = true; # UPO

$wgEmergencyContact = "apache@69.55.229.8";
$wgPasswordSender = "apache@69.55.229.8";

$wgEnotifUserTalk = false; # UPO
$wgEnotifWatchlist = false; # UPO
$wgEmailAuthentication = true;

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost";
$wgDBname = "my_wiki";
$wgDBuser = "root";
$wgDBpassword = "";

# MySQL specific settings
$wgDBprefix = "";

# MySQL table options to use during installation or update
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";

# Experimental charset support for MySQL 5.0.
$wgDBmysql5 = false;

## Shared memory settings
$wgMainCacheType = CACHE_NONE;
$wgMemCachedServers = array();

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads = false;
#$wgUseImageMagick = true;
#$wgImageMagickConvertCommand = "/usr/bin/convert";

# InstantCommons allows wiki to use images from http://commons.wikimedia.org
$wgUseInstantCommons = false;

## If you use ImageMagick (or any other shell command) on a
## Linux server, this will need to be set to the name of an
## available UTF-8 locale
$wgShellLocale = "en_US.utf8";

## If you want to use image uploads under safe mode,
## create the directories images/archive, images/thumb and
## images/temp, and make them all writable. Then uncomment
## this, if it's not already uncommented:
#$wgHashedUploadDirectory = false;

## Set $wgCacheDirectory to a writable directory on the web server
## to make your wiki go slightly faster. The directory should not
## be publically accessible from the web.
#$wgCacheDirectory = "$IP/cache";

# Site language code, should be one of the list in ./languages/Names.php
$wgLanguageCode = "en";

$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";

# Site upgrade key. Must be set to a string (default provided) to turn on the
# web installer while LocalSettings.php is in place
$wgUpgradeKey = "3196710f4a7d7332";

## Default skin: you can change the default skin. Use the internal symbolic
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
$wgDefaultSkin = "vector";

## For attaching licensing metadata to pages, and displaying an
## appropriate copyright notice / icon. GNU Free Documentation
## License and Creative Commons licenses are supported so far.
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
$wgRightsUrl = "";
$wgRightsText = "";
$wgRightsIcon = "";

# Path to the GNU diff3 utility. Used for conflict resolution.
$wgDiff3 = "/usr/bin/diff3";

# Query string length limit for ResourceLoader. You should only set this if
# your web server has a query string length limit (then set it to that limit),
# or if you have suhosin.get.max_value_length set in php.ini (then set it to
# that value)
$wgResourceLoaderMaxQueryLength = -1;

# End of automatically generated settings.
# Add more configuration options below.

</pre>

== copy/backup wiki ==
on main/primary wiki:
<pre>
/usr/local/etc/rc.d/mysql-server stop
ssh 69.55.230.18 "/etc/init.d/mysql stop"
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/
/usr/local/etc/rc.d/mysql-server start
ssh 69.55.230.18 "/etc/init.d/mysql start"

</pre>

Infrastructure Machines

2013-03-25T02:39:53Z

76.88.21.129: /* mediawiki setup */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>

cd /usr/ports/net/rsync
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com

cat /usr/local/www/wiki/LocalSettings.php

<?php
# This file was automatically generated by the MediaWiki 1.19.2
# installer. If you make manual changes, please keep track in case you
# need to recreate them later.
#
# See includes/DefaultSettings.php for all configurable settings
# and their default values, but don't forget to make changes in _this_
# file, not there.
#
# Further documentation for configuration settings may be found at:
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# Protect against web entry
if ( !defined( 'MEDIAWIKI' ) ) {
exit;
}

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

$wgSitename = "JCWiki";

## The URL base path to the directory containing the wiki;
## defaults for all runtime URL paths are based off of this.
## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
$wgScriptPath = "";
$wgScriptExtension = ".php";

## The protocol and server name to use in fully-qualified URLs
$wgServer = "https://69.55.229.8";

## The relative URL path to the skins directory
$wgStylePath = "$wgScriptPath/skins";

## The relative URL path to the logo. Make sure you change this from the default,
## or else you'll overwrite your logo when you upgrade!
#$wgLogo = "$wgStylePath/common/images/wiki.png";
$wgLogo = "$wgStylePath/common/images/jclogo.gif";

## UPO means: this is also a user preference option

$wgEnableEmail = true;
$wgEnableUserEmail = true; # UPO

$wgEmergencyContact = "apache@69.55.229.8";
$wgPasswordSender = "apache@69.55.229.8";

$wgEnotifUserTalk = false; # UPO
$wgEnotifWatchlist = false; # UPO
$wgEmailAuthentication = true;

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost";
$wgDBname = "my_wiki";
$wgDBuser = "root";
$wgDBpassword = "";

# MySQL specific settings
$wgDBprefix = "";

# MySQL table options to use during installation or update
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";

# Experimental charset support for MySQL 5.0.
$wgDBmysql5 = false;

## Shared memory settings
$wgMainCacheType = CACHE_NONE;
$wgMemCachedServers = array();

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads = false;
#$wgUseImageMagick = true;
#$wgImageMagickConvertCommand = "/usr/bin/convert";

# InstantCommons allows wiki to use images from http://commons.wikimedia.org
$wgUseInstantCommons = false;

## If you use ImageMagick (or any other shell command) on a
## Linux server, this will need to be set to the name of an
## available UTF-8 locale
$wgShellLocale = "en_US.utf8";

## If you want to use image uploads under safe mode,
## create the directories images/archive, images/thumb and
## images/temp, and make them all writable. Then uncomment
## this, if it's not already uncommented:
#$wgHashedUploadDirectory = false;

## Set $wgCacheDirectory to a writable directory on the web server
## to make your wiki go slightly faster. The directory should not
## be publically accessible from the web.
#$wgCacheDirectory = "$IP/cache";

# Site language code, should be one of the list in ./languages/Names.php
$wgLanguageCode = "en";

$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";

# Site upgrade key. Must be set to a string (default provided) to turn on the
# web installer while LocalSettings.php is in place
$wgUpgradeKey = "3196710f4a7d7332";

## Default skin: you can change the default skin. Use the internal symbolic
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
$wgDefaultSkin = "vector";

## For attaching licensing metadata to pages, and displaying an
## appropriate copyright notice / icon. GNU Free Documentation
## License and Creative Commons licenses are supported so far.
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
$wgRightsUrl = "";
$wgRightsText = "";
$wgRightsIcon = "";

# Path to the GNU diff3 utility. Used for conflict resolution.
$wgDiff3 = "/usr/bin/diff3";

# Query string length limit for ResourceLoader. You should only set this if
# your web server has a query string length limit (then set it to that limit),
# or if you have suhosin.get.max_value_length set in php.ini (then set it to
# that value)
$wgResourceLoaderMaxQueryLength = -1;

# End of automatically generated settings.
# Add more configuration options below.

</pre>

== copy/backup wiki ==
on main/primary wiki:
<pre>
/usr/local/etc/rc.d/mysql-server stop
ssh 69.55.230.18 "/etc/init.d/mysql stop"
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
rsync -av /var/db/mysql/ib* 69.55.230.18:/var/lib/mysql/
/usr/local/etc/rc.d/mysql-server start
ssh 69.55.230.18 "/etc/init.d/mysql start"

</pre>

Infrastructure Machines

2013-03-25T02:30:44Z

76.88.21.129: /* mediawiki setup */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>

cd /usr/ports/net/rsync
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com

cat /usr/local/www/wiki/LocalSettings.php

<?php
# This file was automatically generated by the MediaWiki 1.19.2
# installer. If you make manual changes, please keep track in case you
# need to recreate them later.
#
# See includes/DefaultSettings.php for all configurable settings
# and their default values, but don't forget to make changes in _this_
# file, not there.
#
# Further documentation for configuration settings may be found at:
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# Protect against web entry
if ( !defined( 'MEDIAWIKI' ) ) {
exit;
}

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

$wgSitename = "JCWiki";

## The URL base path to the directory containing the wiki;
## defaults for all runtime URL paths are based off of this.
## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
$wgScriptPath = "";
$wgScriptExtension = ".php";

## The protocol and server name to use in fully-qualified URLs
$wgServer = "https://69.55.229.8";

## The relative URL path to the skins directory
$wgStylePath = "$wgScriptPath/skins";

## The relative URL path to the logo. Make sure you change this from the default,
## or else you'll overwrite your logo when you upgrade!
#$wgLogo = "$wgStylePath/common/images/wiki.png";
$wgLogo = "$wgStylePath/common/images/jclogo.gif";

## UPO means: this is also a user preference option

$wgEnableEmail = true;
$wgEnableUserEmail = true; # UPO

$wgEmergencyContact = "apache@69.55.229.8";
$wgPasswordSender = "apache@69.55.229.8";

$wgEnotifUserTalk = false; # UPO
$wgEnotifWatchlist = false; # UPO
$wgEmailAuthentication = true;

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost";
$wgDBname = "my_wiki";
$wgDBuser = "root";
$wgDBpassword = "";

# MySQL specific settings
$wgDBprefix = "";

# MySQL table options to use during installation or update
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";

# Experimental charset support for MySQL 5.0.
$wgDBmysql5 = false;

## Shared memory settings
$wgMainCacheType = CACHE_NONE;
$wgMemCachedServers = array();

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads = false;
#$wgUseImageMagick = true;
#$wgImageMagickConvertCommand = "/usr/bin/convert";

# InstantCommons allows wiki to use images from http://commons.wikimedia.org
$wgUseInstantCommons = false;

## If you use ImageMagick (or any other shell command) on a
## Linux server, this will need to be set to the name of an
## available UTF-8 locale
$wgShellLocale = "en_US.utf8";

## If you want to use image uploads under safe mode,
## create the directories images/archive, images/thumb and
## images/temp, and make them all writable. Then uncomment
## this, if it's not already uncommented:
#$wgHashedUploadDirectory = false;

## Set $wgCacheDirectory to a writable directory on the web server
## to make your wiki go slightly faster. The directory should not
## be publically accessible from the web.
#$wgCacheDirectory = "$IP/cache";

# Site language code, should be one of the list in ./languages/Names.php
$wgLanguageCode = "en";

$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";

# Site upgrade key. Must be set to a string (default provided) to turn on the
# web installer while LocalSettings.php is in place
$wgUpgradeKey = "3196710f4a7d7332";

## Default skin: you can change the default skin. Use the internal symbolic
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
$wgDefaultSkin = "vector";

## For attaching licensing metadata to pages, and displaying an
## appropriate copyright notice / icon. GNU Free Documentation
## License and Creative Commons licenses are supported so far.
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
$wgRightsUrl = "";
$wgRightsText = "";
$wgRightsIcon = "";

# Path to the GNU diff3 utility. Used for conflict resolution.
$wgDiff3 = "/usr/bin/diff3";

# Query string length limit for ResourceLoader. You should only set this if
# your web server has a query string length limit (then set it to that limit),
# or if you have suhosin.get.max_value_length set in php.ini (then set it to
# that value)
$wgResourceLoaderMaxQueryLength = -1;

# End of automatically generated settings.
# Add more configuration options below.

</pre>

== copy/backup wiki
on main/primary wiki:
<pre>
/usr/local/etc/rc.d/mysql-server stop
ssh 69.55.230.18 "/etc/init.d/mysql stop"
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
/usr/local/etc/rc.d/mysql-server start
ssh 69.55.230.18 "/etc/init.d/mysql start"

</pre>

Infrastructure Machines

2013-03-25T02:29:42Z

76.88.21.129:

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>

cd /usr/ports/net/rsync
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com

cat /usr/local/www/wiki/LocalSettings.php

<?php
# This file was automatically generated by the MediaWiki 1.19.2
# installer. If you make manual changes, please keep track in case you
# need to recreate them later.
#
# See includes/DefaultSettings.php for all configurable settings
# and their default values, but don't forget to make changes in _this_
# file, not there.
#
# Further documentation for configuration settings may be found at:
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# Protect against web entry
if ( !defined( 'MEDIAWIKI' ) ) {
exit;
}

## Uncomment this to disable output compression
# $wgDisableOutputCompression = true;

$wgSitename = "JCWiki";

## The URL base path to the directory containing the wiki;
## defaults for all runtime URL paths are based off of this.
## For more information on customizing the URLs please see:
## http://www.mediawiki.org/wiki/Manual:Short_URL
$wgScriptPath = "";
$wgScriptExtension = ".php";

## The protocol and server name to use in fully-qualified URLs
$wgServer = "https://69.55.229.8";

## The relative URL path to the skins directory
$wgStylePath = "$wgScriptPath/skins";

## The relative URL path to the logo. Make sure you change this from the default,
## or else you'll overwrite your logo when you upgrade!
$wgLogo = "$wgStylePath/common/images/wiki.png";

## UPO means: this is also a user preference option

$wgEnableEmail = true;
$wgEnableUserEmail = true; # UPO

$wgEmergencyContact = "apache@69.55.229.8";
$wgPasswordSender = "apache@69.55.229.8";

$wgEnotifUserTalk = false; # UPO
$wgEnotifWatchlist = false; # UPO
$wgEmailAuthentication = true;

## Database settings
$wgDBtype = "mysql";
$wgDBserver = "localhost";
$wgDBname = "my_wiki";
$wgDBuser = "root";
$wgDBpassword = "";

# MySQL specific settings
$wgDBprefix = "";

# MySQL table options to use during installation or update
$wgDBTableOptions = "ENGINE=InnoDB, DEFAULT CHARSET=binary";

# Experimental charset support for MySQL 5.0.
$wgDBmysql5 = false;

## Shared memory settings
$wgMainCacheType = CACHE_NONE;
$wgMemCachedServers = array();

## To enable image uploads, make sure the 'images' directory
## is writable, then set this to true:
$wgEnableUploads = false;
#$wgUseImageMagick = true;
#$wgImageMagickConvertCommand = "/usr/bin/convert";

# InstantCommons allows wiki to use images from http://commons.wikimedia.org
$wgUseInstantCommons = false;

## If you use ImageMagick (or any other shell command) on a
## Linux server, this will need to be set to the name of an
## available UTF-8 locale
$wgShellLocale = "en_US.utf8";

## If you want to use image uploads under safe mode,
## create the directories images/archive, images/thumb and
## images/temp, and make them all writable. Then uncomment
## this, if it's not already uncommented:
#$wgHashedUploadDirectory = false;

## Set $wgCacheDirectory to a writable directory on the web server
## to make your wiki go slightly faster. The directory should not
## be publically accessible from the web.
#$wgCacheDirectory = "$IP/cache";

# Site language code, should be one of the list in ./languages/Names.php
$wgLanguageCode = "en";

$wgSecretKey = "abc699ef26890b49b4055430f8ebbd25e84cce21a7e53aeaec4d4313af4c9739";

# Site upgrade key. Must be set to a string (default provided) to turn on the
# web installer while LocalSettings.php is in place
$wgUpgradeKey = "3196710f4a7d7332";

## Default skin: you can change the default skin. Use the internal symbolic
## names, ie 'standard', 'nostalgia', 'cologneblue', 'monobook', 'vector':
$wgDefaultSkin = "vector";

## For attaching licensing metadata to pages, and displaying an
## appropriate copyright notice / icon. GNU Free Documentation
## License and Creative Commons licenses are supported so far.
$wgRightsPage = ""; # Set to the title of a wiki page that describes your license/copyright
$wgRightsUrl = "";
$wgRightsText = "";
$wgRightsIcon = "";

# Path to the GNU diff3 utility. Used for conflict resolution.
$wgDiff3 = "/usr/bin/diff3";

# Query string length limit for ResourceLoader. You should only set this if
# your web server has a query string length limit (then set it to that limit),
# or if you have suhosin.get.max_value_length set in php.ini (then set it to
# that value)
$wgResourceLoaderMaxQueryLength = -1;

# End of automatically generated settings.
# Add more configuration options below.

</pre>

== copy/backup wiki
on main/primary wiki:
<pre>
/usr/local/etc/rc.d/mysql-server stop
ssh 69.55.230.18 "/etc/init.d/mysql stop"
rsync -av /var/db/mysql/my_wiki/ 69.55.230.18:/var/lib/mysql/my_wiki/
/usr/local/etc/rc.d/mysql-server start
ssh 69.55.230.18 "/etc/init.d/mysql start"

</pre>

Infrastructure Machines

2013-03-25T00:05:45Z

76.88.21.129: /* mediawiki setup */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>
cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php

use mysql (innodb)
wiki name: JCWiki
Support / (mail pass) / support@johncompanies.com
</pre>

Infrastructure Machines

2013-03-24T23:50:20Z

76.88.21.129: /* mediawiki setup */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>
cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

https://69.55.229.8/index.php
</pre>

Infrastructure Machines

2013-03-24T07:13:16Z

76.88.21.129: /* mediawiki setup */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>
cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/lang/php5-extensions
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf
DocumentRoot "/usr/local/www/"

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

<IfModule mod_php5.c>
<FilesMatch "\.ph(p3?|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>
# To re-enable php in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_value engine Off
</Directory>
</IfModule>
</IfModule>

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
DirectoryIndex index.php
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

cat > /usr/local/www/wiki/.htaccess
AuthType Basic
AuthUserFile /usr/local/etc/apache22/wiki.passwd
AuthName wiki
require valid-user
satisfy any

cd /usr/local/etc/apache22
htpasswd -c wiki.passwd admin

</pre>

Infrastructure Machines

2013-03-24T06:50:24Z

76.88.21.129: /* mediawiki setup */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>
cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean
(build apache module)

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

</pre>

Infrastructure Machines

2013-03-24T06:45:16Z

76.88.21.129: /* mediawiki setup */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>
cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

</pre>

Infrastructure Machines

2013-03-24T06:30:04Z

76.88.21.129: /* mediawiki setup */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>
cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/httpd-error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/log/httpd-access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /usr/local/etc/apache22/ssl/server.crt
SSLCertificateKeyFile /usr/local/etc/apache22/ssl/server.key

</VirtualHost>

mkdir ssl
cd ssl

openssl req -days 1999 -new -x509 -nodes -out server.crt -keyout server.key
US
CA
San Diego
johncompanies.com
johncompanies.com
wiki.johncompanies.com
support@johncompanies.com

</pre>

Infrastructure Machines

2013-03-24T03:56:53Z

76.88.21.129: /* Setup jail */

= jails =

== jail1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4.101, Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== jail2 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

= virts =

== quar1 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt9 ==

* Location: castle, cab 3-
* OS: FreeBSD x86
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== virt11 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Dell 2950. 6 SATA/SAS drive bays (3 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI-based, Dell-branded (perc) RAID card.

== virt15 ==

* Location: castle, cab 3-
* OS: CentOS
* Networking: Priv IP: 10.1.4., Pub IP: 69.55.2 1 onboard and 1 PCI
* Hardware: Supermicro caase, Intel mobo. 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two GB (2 x GB) RAID1 arrays running on an LSI megaraid RAID card.

= mail =
== Summary ==
This machine (mail) is the swiss army knife of the company, playing host to many services and functions.

* Location: castle, cab 3-8
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
* Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

== Services Provided ==
* mail
* web
* mysql
* bigbrother server/pager
* snmp

== email ==
This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains
Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:
* dave
* linux
* support
* payments
* sales
* tech1
* info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example:
<pre># more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
{
:0h
FROMANDSUBJECT=|formail -XFrom: -XSubject:

:0fwh
| /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

:0
! -t
}

</pre>

control: <tt>cd /etc/mail; make stop</tt> (stop), <tt>cd /etc/mail; make start</tt> (start)

The following aliases are also in place:

<pre>debian: linux
jobs: info
careers: info
#reboot: 6128102202@txt.att.net
#reboot: 8582298897@vtext.com
reboot: pager
#pager: 8582298897@vtext.com
pager: 4158718324@txt.att.net
tech1on: "| /usr/local/sbin/tech1on.sh"
tech1off: "| /usr/local/sbin/tech1off.sh"</pre>

To change them, edit <tt>/etc/aliases</tt> and then run <tt>newaliases</tt>

Note on tech1: this address was setup as a read-only address to be mirrored on all email coming into support and linux. We set this up so we could easily check support mail via a pop client- popping email locks out the user in pine so checking support/linux directly via pop was not an option. When checking and responding to email that comes into tech1, care should be taken to make sure it is sent as/under an address other than tech1. This is cause tech1 is not monitored by support staff as closely as email to support/linux. Further, the tech on call may not be checking tech1. Lastly, because of the nature of the copying, you will sometimes notice certain automated email/notices are received 2x in support- this is because of/related to the tech1 mirror.

To enable it (on mail, run):

<tt>~support/tech1on.sh</tt>

To disable

<tt>~support/tech1off.sh</tt>

Or via email:

<tt>tech1on@johncompanies.com
tech1off@johncompanies.com</tt>

== web ==

See [[Management_System_/_Public_Website_/_Signup|Management System / Public Website / Signup]]

== mysql ==
mysql 4.1.22 is running on port 3306

* datadir: <tt>/mnt/data1/db/mysql/</tt>
* config: <tt>/etc/my.cnf</tt>
* database: <tt>jc</tt>
* control: <tt>/usr/local/etc/rc.d/mysql-server.sh stop</tt> (stop), <tt>/usr/local/etc/rc.d/mysql-server.sh start</tt> (start)

== bigbrother ==
There is a client running on mail (which monitors the services running on mail and mail itself), installed under <tt>/usr/home/bb/bbc1.9e-btf</tt> 
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under <tt>/usr/home/bb/bbsrc/bb1.9i-btf</tt>

Both are running under the user <tt>bb</tt>

Refer to [[BigBrother]] for more about use.

== DNS (ns1c.johncompanies.com) ==
ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from <tt>/mnt/data1/ns1c-dir</tt>

See [[DNS]] for more details

== Usage and Notes ==
* always mounted to backup1 and backup2 via nfs:
<pre>backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)
</pre>

== Cronjobs ==
* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1
Gathers up data for our mrtg/load graphs

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/
Gathers up data from i2b servers for our mrtg/load graphs

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg
Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done
Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack
Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth

1 0 * * * /usr/local/etc/rsync.backup
Nightly backup script

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update
Public web traffic stats

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*
Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl
Monthly bandwidth overage report

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl
Updates mgmt with bb monitoring issues

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl
Emails customers reminding them of upcoming shutdown date

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl
Emails customers who have invoices and are set to auto-email (currently no customer gets these)

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl
Checking that we are properly replicating (mysql) traffic data from bwdb to backup1

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl
Removed old traffic data from the traffic database (running on backup1)

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done
Secure credit card data: set root-read-only

25 0 * * * /usr/local/www/cronjobs/biller.pl
Enters service charges in customer billing ledgers

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl
Looks for customers with balance due and active credit card on file, prepares a payflow batch

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl
Tries to collect ccard funds for items in payflow batch - communicates with payflow

15 13 * * * /usr/local/www/cronjobs/pb_batch_gather.pl
Looks for customers with balance due and active paypal billing agreement on file, prepares a paypal batch

15 14 * * * /usr/local/www/cronjobs/pb_batch_process.pl
Tries to collect paypal funds for items in paypal batch - communicates with paypal

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl
Emails customers in arrears, reminding them to pay

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null
Reminds us to archive sent mail

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys
Backup data on isys

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

= nat =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions. Further, it's ip runs in a special block which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.1, Pub IPs: 69.55.233.195, 69.55.233.196, 69.55.233.197, 69.55.233.198, 69.55.233.199. 1 onboard and 1 PCI
* Hardware: Custom 1U. single power supply.
* Drives: one 8 GB IDE drive

== Services Provided ==
* nat

== nat control ==
All rules are contained in and look like:

<pre>cat /etc/ipnat.rules
# www (was 69.55.230.12)
# virt19
#bimap fxp0 10.1.4.209/32 -> 69.55.233.198/32
# virt18
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
# virt13
#bimap fxp0 10.1.4.213/32 -> 69.55.233.196/32
# virt12
#bimap fxp0 10.1.4.212/32 -> 69.55.233.196/32
# virt17
bimap fxp0 10.1.4.217/32 -> 69.55.233.196/32
# virt11
#bimap fxp0 10.1.4.211/32 -> 69.55.233.196/32
# ASA
#bimap fxp0 10.1.4.172/32 -> 69.55.233.196/32
# P1A
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
#bimap fxp0 10.1.4.238/32 -> 69.55.233.197/32
# developer (was 69.55.230.17)
# jail2
#bimap fxp0 10.1.4.232/32 -> 69.55.233.198/32
# jail8
#bimap fxp0 10.1.4.238/32 -> 69.55.233.198/32
# jail9
#bimap fxp0 10.1.4.239/32 -> 69.55.233.198/32
# POLL
#BIMAP EM0 10.1.6.134/32 -> 69.55.230.20/32
# 1U SUN
#BIMAP EM0 10.1.4.4/32 -> 69.55.227.46/32
# ??
#BIMAP EM0 10.1.6.3/32 -> 69.55.230.100/32
# random machine
#bimap fxp0 10.1.6.13/32 -> 69.55.233.199/32
#bimap fxp0 10.1.4.232/32 -> 69.55.233.199/32
# OFFICE OUTBOUND TRAFFIC
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp
#map fxp0 10.1.6.0/24 -> 0.0.0.0/32</pre>

A simple entry looks like:
bimap fxp0 10.1.4.240/32 -> 69.55.233.197/32
Which essentially means make private IP 10.1.4.240 reachable on 69.55.233.197 and allow 10.1.4.240 to communicate with the public internet via 69.55.233.197

To reload new rule config:
ipnat -C -F -f /etc/ipnat.rules

You may want to setup natting, as above, when you need to reach a DRAC card's web interface, wherin the DRAC card only has a private IP.

= nat2 =
== Summary ==
This is the main machine to which we ssh and runs all our screen sessions at i2b, and runs ns3c (this is kind of the what mail is to castle). Further, it's ip runs in IP space provided by i2b: 66.181.18.1 - 66.181.18.30, which is not routed through the firewall and this is somewhat immune to DoS attacks which hobble our firewall. Lastly, it acts as a nat server for certain/random devices on the private network.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.1, Pub IPs: 69.55.229.2, 69.55.229.3, 66.181.18.4, 66.181.18.5, 66.181.18.6, 66.181.18.7, 66.181.18.8, 66.181.18.9, 66.181.18.10, 66.181.18.11, 66.181.18.12, 66.181.18.13, 66.181.18.14 1 onboard and 1 PCI
* Hardware: Custom 2U. 6 drive bays, non-hot-swappable. single power supply.
* Drives: one 150 GB (2 x 150GB) RAID1 array running on a 3ware 8006 RAID card.

== Services Provided ==
* nat
* bigbrother
* ns3c (jail)
* ntp

== nat config ==
Here's what's currently nat'd on nat2:
<pre>cat /etc/ipnat.rules
# sample entry
#ATS-9
bimap em0 10.1.2.79/32 -> 66.181.18.14/32
#ATS-8
bimap em0 10.1.2.78/32 -> 66.181.18.13/32
#ATS-7
bimap em0 10.1.2.77/32 -> 66.181.18.12/32
#ATS-6
bimap em0 10.1.2.76/32 -> 66.181.18.6/32
#ATS-5
bimap em0 10.1.2.75/32 -> 66.181.18.7/32
#ATS-4
bimap em0 10.1.2.74/32 -> 66.181.18.8/32
#ATS-3
bimap em0 10.1.2.73/32 -> 66.181.18.9/32
#ATS-2
bimap em0 10.1.2.72/32 -> 66.181.18.10/32
#ATS-1
bimap em0 10.1.2.71/32 -> 66.181.18.11/32
#bwdb2
bimap em0 10.1.2.4/32 -> 66.181.18.5/32

# spare

map em0 10.1.2.0/24 -> 0.0.0.0/32 proxy port ftp ftp/tcp

#bimap fxp0 10.1.6.49/32 -> 10.1.1.2/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32
</pre>

== build ==

* partition map:
<pre>/ 512m
swap 1G
/var 256m
/tmp 256m
/usr 5g
/mnt/data1 ~</pre>

* edit /etc/make.conf
<pre>echo "WITHOUT_X11=yes \
KERNCONF=nat2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

* put key in authorized_keys on backup3
<pre>cd
ssh-keygen -t dsa -b 1024</pre>
(default location, leave password blank)

<pre>cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
<pre>cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null</pre>

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null</pre>

* configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/nat2-6.4 ./nat2

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
make installworld
mergemaster -i
</pre>

* populate /etc/rc.conf with IPs and NFS settings
<pre>vi /etc/rc.conf

hostname="nat2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
xntpd_flags="-A -p /var/run/ntpd.pid"

nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_em0="inet 10.1.6.50 netmask 255.255.255.0"
#ifconfig_em0="inet 69.55.229.2 netmask 255.255.255.0"
#ifconfig_em0_alias0="inet 69.55.229.229 netmask 255.255.255.255"
ifconfig_fxp0="inet 69.55.229.2 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 69.55.229.3 netmask 255.255.255.255"
ifconfig_fxp1="inet 10.1.2.1 netmask 255.255.255.0"
defaultrouter="10.1.6.1"
#defaultrouter=" 66.181.14.250"
snmpd_enable="YES"
ipnat_enable="YES"
ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.1"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
fetch http://3ware.com/download/Escalade9690SA-Series/9.5.3/tw_cli-freebsd-x86-9.5.3.tgz
tar xzf tw_cli-freebsd-x86-9.5.3.tgz
rm tw_cli-freebsd-x86-9.5.3.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

* install rsync from ports
<pre>cd /usr/ports/net/rsync
make install clean</pre>
choose default options

* install perl from ports
<pre>cd /usr/ports/lang/perl5.8
make install clean</pre>

* install screen from ports
<pre>cd /usr/ports/sysutils/screen
make install clean</pre>

* install bb client
<pre>adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar</pre>
edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
<pre>echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 nat2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="nat2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh
</pre>

Punch a hole in the firewall to allow it to communicate with bb monitor (probably already exists):

ipfw add 96 allow ip from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail:
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
64.163.14.48 nat2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp
<pre>echo "server 69.55.230.2
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org
server 66.187.233.4
server 217.204.76.170
server 64.112.189.11
server 66.69.112.130
server 80.85.129.25
server 80.237.234.15
server 130.60.7.44
server 134.99.176.3
server 198.144.202.250
server 202.74.170.194
server 204.17.42.199
server 204.87.183.6
server 213.15.3.1
server 213.239.178.33
server 217.114.97.97
server 69.55.230.2" > /etc/ntp.conf</pre>

<pre>/usr/sbin/ntpd -A -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups, nfs mount

<pre>mkdir /backup3
echo 'backup3:/data /backup3 nfs rw,bg 0 0' >> /etc/fstab

echo '#\!/bin/sh\
backupdir=/data/nat2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config</pre>

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/nat2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

<pre>scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3</pre>

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

* edit sshd_config for security
<pre>vi /etc/ssh/sshd_config
ListenAddress 66.181.18.1
ListenAddress 69.55.229.2
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`</pre>

* raid chk

<pre>cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}</pre>

* netflow stuff
add crontab entries
<pre>crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*</pre>

<pre>scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/</pre>

other binaries

<pre>scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~</pre>

* add nat rules
<pre>vi /etc/ipnat.rules
# sample entry
bimap fxp0 10.1.6.70/32 -> 10.1.6.59/32
#bimap fxp0 10.1.4.208/32 -> 69.55.233.196/32

ipnat -C -f /etc/ipnat.rules</pre>

* shell for user
<pre>cp /root/.cshrc ~user/
vi ~user/</pre>
change # to $

* mrtg

<pre>cd /usr/ports/net-mgmt/mrtg
make install clean</pre>
(no FONTCONFIG, v3)

this didn't work cause of libtool incompat

so manually moved files:

<pre>scp /usr/local/bin/cfgmaker user@nat2:/usr/local/bin/cfgmaker
scp /usr/local/lib/perl5/site_perl/5.6.1/MRTG_lib.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_util.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/BER.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/lib/perl5/site_perl/5.6.1/SNMP_Session.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/
scp /usr/local/bin/mrtg root@nat2:/usr/local/bin/mrtg
scp /usr/local/lib/perl5/site_perl/5.6.1/locales_mrtg.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/locales_mrtg.pm
scp /usr/local/bin/rrdtool root@nat2:/usr/local/bin/rrdtool
scp /usr/local/lib/perl5/site_perl/5.6.1/mach/RRDs.pm root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/RRDs.pm
rsync -av /usr/local/lib/perl5/site_perl/5.6.1/mach/auto/RRDs/ root@nat2:/usr/local/lib/perl5/site_perl/5.8.9/mach/auto/RRDs/
scp /usr/lib/libz.so.2 root@nat2:/usr/lib/libz.so.2
scp /usr/lib/libm.so.2 root@nat2:/usr/lib/libm.so.2
rsync -av /usr/local/lib/librrd* root@nat2:/usr/local/lib/
scp /usr/lib/libc.so.4 root@nat2:/usr/lib/libc.so.4

rsync -av /usr/ports/net/rrdtool root@nat2:/usr/ports/net
cd /usr/ports/net/rrdtool
make install

mkdir -p /mnt/data1/mrtg/data
scp /usr/local/www/mgmt/mrtg/template.pl root@nat2:/mnt/data1/mrtg/
scp /usr/local/www/mgmt/mrtg/host.pl root@nat2:/mnt/data1/mrtg/

cfgmaker --if-template=template.pl --show-op-down --global "options[_]: growright,bits" --global 'WorkDir: /mnt/data1/mrtg/data' --global 'Interval: 1' --global 'LogFormat: rrdtool' --global 'PathAdd: /usr/local/bin' --global 'LibAdd: /usr/local/lib' --host-template=host.pl jc292401@10.1.2.50 --output=switch-p20.cfg

cat > /mnt/data1/mrtg/mrtg.sh
#!/bin/sh
/usr/local/bin/mrtg /mnt/data1/mrtg/switch-p20.cfg

chmod 0700 /mnt/data1/mrtg/mrtg.sh

crontab -e
* * * * * /mnt/data1/mrtg/mrtg.sh 2>&1 > /dev/null</pre>

* snmp firewall block
<pre>cat > /usr/local/etc/rc.d/boot.sh
ipfw add 10 allow udp from 69.55.230.2 to any 161
ipfw add 10 allow udp from 10.1.2.1 to any 161
ipfw add 11 deny udp from any to any 161
chmod 0700 /usr/local/etc/rc.d/boot.sh</pre>

= bwdb =
== Summary ==
This machine tracks and stores network traffic (netflow) at castle. It is our means to monitor customer bandwidth usage.

* Location: castle, cab 3-7
* OS: FreeBSD 4.10 x86
* Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 1U. Single power supply.
* Drives: one 250 GB (2 x 250GB) RAID1 array running on a Promise IDE RAID card.

== Services Provided ==
* netflow
* mysql
* bigbrother
* snmp

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, inserting the data into the local mysql database:
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl

== mysql ==

The database storing all the traffic data is named <tt>traffic</tt>
Tables:
<pre>mysql> show tables;
+---------------------------+
| Tables_in_traffic |
+---------------------------+
| dailyIpTotals_69_55_224 |
| dailyIpTotals_69_55_225 |
| dailyIpTotals_69_55_226 |
| dailyIpTotals_69_55_227 |
| dailyIpTotals_69_55_228 |
| dailyIpTotals_69_55_229 |
| dailyIpTotals_69_55_230 |
| dailyIpTotals_69_55_231 |
| dailyIpTotals_69_55_232 |
| dailyIpTotals_69_55_233 |
| dailyIpTotals_69_55_234 |
| dailyIpTotals_69_55_235 |
| dailyIpTotals_69_55_236 |
| dailyIpTotals_69_55_237 |
| dailyIpTotals_69_55_238 |
| dailyIpTotals_69_55_239 |
| dailyPortTotals_69_55_224 |
| dailyPortTotals_69_55_225 |
| dailyPortTotals_69_55_226 |
| dailyPortTotals_69_55_227 |
| dailyPortTotals_69_55_228 |
| dailyPortTotals_69_55_229 |
| dailyPortTotals_69_55_230 |
| dailyPortTotals_69_55_231 |
| dailyPortTotals_69_55_232 |
| dailyPortTotals_69_55_233 |
| dailyPortTotals_69_55_234 |
| dailyPortTotals_69_55_235 |
| dailyPortTotals_69_55_236 |
| dailyPortTotals_69_55_237 |
| dailyPortTotals_69_55_238 |
| dailyPortTotals_69_55_239 |
| ipTotals_69_55_224 |
| ipTotals_69_55_225 |
| ipTotals_69_55_226 |
| ipTotals_69_55_227 |
| ipTotals_69_55_228 |
| ipTotals_69_55_229 |
| ipTotals_69_55_230 |
| ipTotals_69_55_231 |
| ipTotals_69_55_232 |
| ipTotals_69_55_233 |
| ipTotals_69_55_234 |
| ipTotals_69_55_235 |
| ipTotals_69_55_236 |
| ipTotals_69_55_237 |
| ipTotals_69_55_238 |
| ipTotals_69_55_239 |
| portTotals_69_55_224 |
| portTotals_69_55_225 |
| portTotals_69_55_226 |
| portTotals_69_55_227 |
| portTotals_69_55_228 |
| portTotals_69_55_229 |
| portTotals_69_55_230 |
| portTotals_69_55_231 |
| portTotals_69_55_232 |
| portTotals_69_55_233 |
| portTotals_69_55_234 |
| portTotals_69_55_235 |
| portTotals_69_55_236 |
| portTotals_69_55_237 |
| portTotals_69_55_238 |
| portTotals_69_55_239 |
+---------------------------+
</pre>

So as you see we store each class-C block in its own table, for efficiency. Further, we store and organize data in 4 ways: "daily" tables and 15-minute granularity tables, and for each of those we track simple IP traffic and port-specific traffic. The daily tables contains 2 entries (one for each direction) for each IP for each day. For the current day, the row data is incremented as the day goes on.

<pre>mysql> describe dailyIpTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(23) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(12) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+

mysql> select * from dailyIpTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 6955224194-20100917-1
date: 2010-09-17
ip: 69.55.224.194
direction: 1
octets: 8821
packets: 91
</pre>

The <tt>id</tt> is a unique identifier (key), <tt>direction</tt> indicates incoming or outgoing traffic (outbound = 2, inbound = 1), <tt>octets</tt> are the amount of traffic in kilobytes, and <tt>packets</tt> is the total number of packets.

The 15-minute table has similar information, but it's organized in 15 minute increments:

<pre>mysql> describe ipTotals_69_55_224;
+-----------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+-------+
| date | datetime | YES | | NULL | |
| ip | char(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| octets | bigint(20) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+------------+------+-----+---------+-------+

mysql> select * from ipTotals_69_55_224 limit 2\G
*************************** 1. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.13
direction: 1
octets: 288
packets: 6
*************************** 2. row ***************************
date: 2010-01-11 19:30:00
ip: 69.55.224.12
direction: 1
octets: 216
packets: 4</pre>

So for a given IP, there will be 192 rows in a given day: 4 rows per hour, *2 for 2 directions, *24 for 24hours in a day. Obviously this table is large which is why we broke it down into a daily table for quick, easy, daily-summary access.

That covers the simple traffic tabulation tables. We also track traffic by port:

<pre>mysql> describe dailyPortTotals_69_55_224;
+-----------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| id | varchar(28) | | PRI | | |
| date | date | YES | | NULL | |
| ip | varchar(15) | YES | MUL | NULL | |
| direction | tinyint(1) | YES | | NULL | |
| protocol | smallint(3) | YES | | NULL | |
| port | int(11) | YES | | NULL | |
| octets | bigint(11) | YES | | NULL | |
| packets | int(11) | YES | | NULL | |
+-----------+-------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from dailyPortTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
id: 695522496-20091218-1-6-23
date: 2009-12-18
ip: 69.55.224.96
direction: 1
protocol: 6
port: 23
octets: 1796
packets: 30

mysql> select * from portTotals_69_55_224 limit 1\G
*************************** 1. row ***************************
date: 2010-09-07 18:45:00
ip: 69.55.224.254
direction: 1
protocol: 6
port: 99999
octets: 144
packets: 3

</pre>

This is largely the same with 2 more additions: <tt>protocol</tt> (1=ICMP, 6=TCP, 17=UDP), and <tt>port</tt> which we set to 99999 if the traffic is return traffic and the port is above 1024. Obviously the potential for number of rows grows quickly when you consider the addition of port and protocol tracking per IP.

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]
* archive data from database
archive_daily.pl 2012 09
This will archive data for the given year and month from the daily summary tables. Generally we want to have a year of history in the database.

archive_15min.pl 2012 09
This will archive data for the given year and month from the 15min-increment tables. Generally, we want to have 6 months of history in the database.

* if space becomes tight, move flow files and exported data to a backup server, both located in <tt>/usr/home/flowbin/archive</tt> and <tt>/usr/home/exported</tt>, respectively

== Slaving ==

If we were going to setup traffic database slaving (we don't do this anymore), perhaps cause the bwdb machine gets busy and it cannot handle traffic requests and netflow, here's how it's done:

On the traffic master:

GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.4.8' IDENTIFIED BY 'qERUG8wf';

in my.cnf:
<pre>bin-log
server-id=1
max_binlog_size=500M
expire_logs_days = 3</pre>

on slave:
in my.cnf:
<pre>server-id = 2
master-host = 10.1.4.203
master-user = repl
master-password = qERUG8wf
master-connect-retry=60
replicate-wild-do-table=traffic.daily%
max_relay_log_size=500M
expire_logs_days = 3

replicate-wild-do-table=traffic.%</pre>

on master:
touch /usr/home/working/.lock
(make sure processflows not running)

<pre>FLUSH TABLES WITH READ LOCK;
cd /usr/home/database/traffic
tar -czf mysql-traffic-snapshot.tgz ./daily*
(~1G)
SHOW MASTER STATUS;
+-----------------+-----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-----------------+-----------+--------------+------------------+
| bwdb-bin.000039 | 154432615 | | |
+-----------------+-----------+--------------+------------------+

(write down info)
UNLOCK TABLES;
scp mysql-traffic-snapshot.tgz 10.1.4.5:/mnt/data1/db/mysql/traffic/</pre>

on slave:
<pre>mkdir /mnt/data1/db/mysql/traffic
cd /mnt/data1/db/mysql/traffic/
tar xzvf mysql-traffic-snapshot.tgz
(restart mysql)
CHANGE MASTER TO MASTER_HOST='10.1.4.203',MASTER_USER='repl',MASTER_PASSWORD='qERUG8wf',MASTER_LOG_FILE='bwdb-bin.000059',MASTER_LOG_POS=482502186;
START SLAVE;</pre>

<pre>cd /usr/home/database/traffic
scp *</pre>

<pre>optimize table dailyPortTotals_69_55_224;
optimize table dailyPortTotals_69_55_225;
optimize table dailyPortTotals_69_55_226;
optimize table dailyPortTotals_69_55_227;
optimize table dailyPortTotals_69_55_228;
optimize table dailyPortTotals_69_55_229;
optimize table dailyPortTotals_69_55_230;
optimize table dailyPortTotals_69_55_231;
optimize table dailyPortTotals_69_55_232;
optimize table dailyPortTotals_69_55_233;
optimize table dailyPortTotals_69_55_234;
optimize table dailyPortTotals_69_55_235;
optimize table dailyPortTotals_69_55_236;
optimize table dailyPortTotals_69_55_237;
optimize table dailyPortTotals_69_55_238;
optimize table dailyPortTotals_69_55_239;</pre>

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===

Install FreeBSD 8.3 amd64

* partition map:
<pre>/ 500m
swap 4096m
/var 256m
/tmp 256m
/usr ~</pre>

* edit /etc/make.conf
Castle:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

i2b:
<pre>echo "WITHOUT_X11=yes \
KERNCONF=bwdb2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf</pre>

* add settings to /boot/loader.conf and /boot.config

<pre>echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf</pre>

* turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyu0, change type to vt100:
<pre>vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyu0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1</pre>

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

* populate hosts
i2b:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

castle:
<pre>echo "10.1.4.3 backup2 backup2.johncompanies.com" >> /etc/hosts
echo "10.1.4.8 backup1 backup1.johncompanies.com" >> /etc/hosts
echo "10.1.4.4 mail mail.johncompanies.com" >> /etc/hosts
</pre>

* put key in authorized_keys on backup1 and backup2
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

castle:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

i2b:
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup2 and backup1 (and backup3 if at i2b) without getting a login prompt

ssh backup1 hostname
ssh backup2 hostname

* edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
<pre>alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
</pre>

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

* install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' support@johncompanies.com < /dev/null

* get latest sources for this release:
<pre>cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' support@johncompanies.com < /dev/null</pre>

* configure new kernel

cd /usr/src/sys/amd64/conf
scp backup2:/mnt/data4/build/freebsd/kern_config-bwdb-8.3-amd64 ./bwdb

Edit config and change name:
vi bwdb
ident bwdb

* build, install kernel and world

<pre>cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' support@johncompanies.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i</pre>

* populate /etc/rc.conf with IPs and NFS settings

castle:
<pre>vi /etc/rc.conf

hostname="bwdb.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.4.203 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.4.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.4.203"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

i2b:
<pre>vi /etc/rc.conf

hostname="bwdb2.johncompanies.com"
kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
ifconfig_fxp0="inet 10.1.2.4 netmask 255.255.255.0"
ifconfig_em0="up promisc"
defaultrouter="10.1.2.1"
snmpd_enable="YES"

inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.4"
fsck_y_enable="YES"
background_fsck="NO"
sshd_enable="YES"
ipfw_load="YES"</pre>

* reboot. Confirm new kernel is loaded

uname -a

* update ports:
<pre>cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_8_3\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' support@johncompanies.com < /dev/null</pre>

* Install raid mgmt tool

<pre>cd /usr/local/sbin
scp backup2:/d4/build/3ware/tw_cli-freebsd-x86_64-9.5.0.1.tgz .
tar xzf tw_cli-freebsd-x86_64-9.5.0.1.tgz
rm tw_cli-freebsd-x86_64-9.5.0.1.tgz
chmod 0700 tw_cli</pre>

Test:
./tw_cli info c0

Grab raid check script:
scp backup1:/usr/local/sbin/3wraidchk /usr/local/etc

Setup cronjob:
<pre>crontab -e
*/5 * * * * /usr/local/etc/3wraidchk</pre>

* install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

* install perl from ports
cd /usr/ports/lang/perl5.8
make install clean

choose default options

* install bb client

Compiling from source on AMD64 will not work. So, we use a linux-compiled version and rely on linux compat. Linux compat won't install on 8.x - libtool 2.4 need. So, instead we copy(ed) over linux:
rsync -aSHv --exclude=proc --exclude=sys 10.1.4.108:/usr/compat/linux/ /usr/compat/linux/

adduser

Output/response:

<pre>Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes</pre>

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd_linuxcompat.tgz .
tar xzf bb-freebsd_linuxcompat.tgz

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.4.203 bwdb.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

Edit for machine name and private IP.

if this machine is at i2b:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.4 bwdb2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

<pre>vi /home/bb/bbc1.9e-btf/ext/openfiles

MACHINE="bwdb,johncompanies,com" # HAS TO BE IN A,B,C FORM</pre>
Edit for machine name.

Have bb watch for flow-capture, mysql
<pre>cat >> /home/bb/bbc1.9e-btf/etc/bb-proctab
localhost: flow-capture :
localhost: mysqld :</pre>

<pre>cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf
./runbb.sh start
more BBOUT
(look for errors)
exit</pre>

Put in script to start bb @ boot:
echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

If this is at i2b, punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow tcp from 66.181.18.0/27 to 69.55.230.2

* configure bb on mail
<pre>vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
10.1.4.203 bwdb.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit</pre>

* configure ntp server
Castle:
echo "server 10.1.4.1" > /etc/ntp.conf

I2b:
echo "server 10.1.2.1" > /etc/ntp.conf

<pre>/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p</pre>
(confirm it’s able to reach our time server)

<pre>echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh</pre>

* fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

* setup backups
<pre>echo '#\!/bin/sh\
backupdir=/data/bwdb/current\
server=backup1\
\
## ENTRY /etc\
## ENTRY /usr/home/flowbin\
## ENTRY /usr/home/database' > /usr/local/etc/backup.config</pre>

Castle:
setup backup dirs:
ssh backup1 mkdir -p /data/bwdb/current
on backup1, add the system to
vi /usr/local/sbin/snapshot_rotate

I2b:
setup backup dirs:
ssh backup3 mkdir -p /data/bwdb/current
on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

Copy over the backup script:
scp backup2:/d4/bin/freebsd8.x/rsync.backup /usr/local/etc/

Edit rsync.backup and change <tt>config</tt> var to point to correct config file location: <tt>/usr/local/etc/backup.config</tt>

<pre>crontab -e
5 0 * * * /usr/local/etc/rsync.backup</pre>

* make /root/logs
mkdir /root/logs

* edit sshd_config for security

<pre>vi /etc/ssh/sshd_config
ListenAddress 10.1.4.203
PermitRootLogin yes

kill -1 `cat /var/run/sshd.pid`</pre>

Edit for private IP.

* snmp

(Before doing this you may need to take down the firewall and also add to resolv.conf 69.43.143.41)
<pre>
cd /usr/ports/net-mgmt/net-snmp
make install clean
(defaults)

cat >> /etc/rc.conf
snmpd_enable="YES"
snmpd_flags="-a"
snmpd_conffile="/usr/local/share/snmp/snmpd.conf"
snmptrapd_enable="YES"
snmptrapd_flags="-a -p /var/run/snmptrapd.pid"

cat > /usr/local/share/snmp/snmpd.conf
rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.202
</pre>

=== netflow ===

Install flow tools:
<pre>cd /usr/ports/net-mgmt/flow-tools
make install clean</pre>
Defaults.

mkdir /usr/home/flows

Flow start script:
echo "/usr/local/bin/flow-capture -w /usr/home/flows -S5 -N -2 0/10.1.4.203/4444" > /usr/local/etc/rc.d/flow-capture.sh

chmod 0700 /usr/local/etc/rc.d/flow-capture.sh

Edit for private IP.

Netgraph start script:
<pre>
cat > /usr/local/etc/rc.d/netgraph.sh

/usr/sbin/ngctl -f- <<-SEQ
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.1.4.203:4444
SEQ

#/usr/sbin/ngctl -f- <<-SEQ
#shutdown netflow:
#SEQ

chmod 0700 /usr/local/etc/rc.d/netgraph.sh</pre>
Edit for private IP.

Confirm netflow is running after running scripts:
<pre>newbwdb /usr/ports/net-mgmt/flow-tools# /usr/sbin/ngctl
Available commands:
config get or set configuration of node at <path>
connect Connects hook <peerhook> of the node at <relpath> to <hook>
debug Get/set debugging verbosity level
dot Produce a GraphViz (.dot) of the entire netgraph.
help Show command summary or get more help on a specific command
list Show information about all nodes
mkpeer Create and connect a new node to the node at "path"
msg Send a netgraph control message to the node at "path"
name Assign name <name> to the node at <path>
read Read and execute commands from a file
rmhook Disconnect hook "hook" of the node at "path"
show Show information about the node at <path>
shutdown Shutdown the node at <path>
status Get human readable status information from the node at <path>
types Show information about all installed node types
write Send a data packet down the hook named by "hook".
quit Exit program
+ show netflow:
Name: netflow Type: netflow ID: 00000004 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export <unnamed> ksocket 00000005 inet/dgram/udp
out0 em0 ether 00000001 upper
iface0 em0 ether 00000001 lower
+
</pre>

We notice that sometimes flow-capture is failing due to swap exhaustion (even after adding more swap). So we crontab flow-capture to restart (it's ok to start if it's already running, it just quits):

<pre>
crontab -e
#restart flow-capture
*/15 * * * * /usr/local/etc/rc.d/flow-capture.sh
</pre>

==== process flow tools ====

<pre>mkdir /usr/home/flowbin
mkdir /usr/home/working</pre>

Install modules:
<pre>cd /usr/ports/devel/p5-Date-Calc
make install clean
cd /usr/ports/mail/p5-Mail-Sendmail
make install clean</pre>

Queue script:
<pre>
cat > /usr/home/flowbin/queue.pl
#!/usr/bin/perl

use strict;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use date;

my $flowbase = "/usr/home/flows";
#my $flowqueue = "/usr/home/queue";
my $flowqueue = "/usr/home/working";

my ($date, $time) = date::CurrentDateTime();

my $flowdir = mkFlowDir($date);
`mv $flowdir/ft-* $flowqueue`;

if (date::DateWindow($date, $time, $date, "00:00:00", 600)) {
my $newdate = date::AddDays($date, -1);
my $flowdir = mkFlowDir($newdate);
`mv $flowdir/ft-* $flowqueue`;
}

sub mkFlowDir {
my $date = shift;
$date =~ /([0-9]{4}-[0-9]{2})/;
my $yearmonth = $1;
return "$flowbase/$yearmonth/$date";
}
</pre>

Date.pm module:
<pre>
cat > /usr/home/flowbin/date.pm
#!/usr/local/bin/perl
#
# $Header: /usr/cvs/newgw/lib/date.pm,v 1.2 2003/11/24 17:06:02 glenn Exp $
#
# Copyright (c) 2001, 2002, 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
#
# date.pl - Higher level functions written on top of Date::Calc

package date;

use strict;
use Date::Calc qw(:all);

sub DayDiff { #calculate the difference in days from two dates
my $date1 = shift;
my $date2 = shift;
my ($year1, $month1, $day1) = &DateToymd($date1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my $diff = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
return $diff;
}

sub AddDays { #adds specified number of days to the supplied date
my $date = shift;
my $days = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($nyear, $nmonth, $nday) = &Add_Delta_Days($year, $month, $day, $days);
my $ndate = &ymdToDate($nyear, $nmonth, $nday);
return $ndate;
}

sub AddHours { #adds specified number of hours to the supplied date and time
my $date = shift;
my $time = shift;
my $addhours = shift;
my $adddays = 0;
if (abs($addhours / 24) >= 1) {
$adddays = int($addhours / 24);
$addhours -= $adddays * 24;
}
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
$adddays, $addhours, 0, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub AddMinutes {
my $date = shift;
my $time = shift;
my $minutes = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($ny, $nm, $nd, $nh, $nmin, $ns) = &Add_Delta_DHMS($year, $month, $day,
$hour, $minute, $second,
0, 0, $minutes, 0);
my $ndate = &ymdToDate($ny, $nm, $nd);
my $ntime = &hmsToTime($nh, $nmin, $ns);
return $ndate, $ntime;
}

sub CurrentDateTime { #return the current date and time
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
my $date = &ymdToDate($y, $m, $d);
my $time = &hmsToTime($h, $min, $s);
return $date, $time;
}

sub Currentymd { #return the current year, month and day as separate variables
my ($y, $m, $d, $h, $min, $s, $z, $z, $z) = &System_Clock;
return $y, $m, $d;
}

sub DateToymd { #takes a date and returns year, month, day as individual values
my $date = shift;
if ($date =~ /([0-9]{4})-([0-9]{2})-([0-9]{2})/) {
my $day = $3;
my $month = $2;
my $year = $1;
return $year, $month, $day;
}
return undef;
}

sub TimeTohms { #takes a time and return hours minutes and seconds as individual values
my $time = shift;
if ($time =~ /([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/) {
my $hour = $1;
my $minute = $2;
my $second = $3;
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return $hour, $minute, $second;
}
return undef;
}

sub ymdToDate { #takes year, month, day and assembles them into our date format
my $year = shift;
my $month = shift;
my $day = shift;
if (defined($year) && defined($month) && defined ($day)) {
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
return "$year-$month-$day";
}
return undef;
}

sub hmsToTime { #takes hour minute and second and assembles them into our time format
my $hour = shift;
my $minute = shift;
my $second = shift;
if (defined($hour) && defined($minute) && defined ($second)) {
if ($hour !~ /[0-9]{2}/) { $hour = "0$hour"; }
if ($minute !~ /[0-9]{2}/) { $minute = "0$minute"; }
if ($second !~ /[0-9]{2}/) { $second = "0$second"; }
return sprintf ("%02d:%02d:%02d", $hour, $minute, $second);
}
return undef;
}

sub CompareDates { #compares two date and time pairs
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

# &debug("$year1, $month1, $day1, $year2, $month2, $day2");
my $days = &Delta_Days($year1, $month1, $day1, $year2, $month2, $day2);
if ($days > 0) { return 1;}
if ($days < 0) { return -1;}
if ($days == 0) { #same day, compare times
my $seconds1 = $second1 + (60 * $minute1) + (3600 * $hour1);
my $seconds2 = $second2 + (60 * $minute2) + (3600 * $hour2);
if ($seconds1 < $seconds2) { return 1;}
if ($seconds1 > $seconds2) { return -1;}
if ($seconds1 == $seconds2) { return 0;}
}
return undef;
}

sub DateWindow { #compares two date time pairs to see if they are < X seconds apart
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $window = shift;

my ($year1, $month1, $day1) = &DateToymd($date1);
my ($hour1, $minute1, $second1) = &TimeTohms($time1);
my ($year2, $month2, $day2) = &DateToymd($date2);
my ($hour2, $minute2, $second2) = &TimeTohms($time2);

my ($day, $hour, $minute, $second) =
&Delta_DHMS($year1, $month1, $day1, $hour1, $minute1, $second1,
$year2, $month2, $day2, $hour2, $minute2, $second2);
$minute *= 60;
$hour *= 3600;
$day *= 86400;
my $total = $second + $minute + $hour + $day;
if (abs($total) < $window) {
return 1;
}
return 0;
}

sub CheckDateOrder { #takes three dates/times, returns true if they are in chronological order
my $date1 = shift;
my $time1 = shift;
my $date2 = shift;
my $time2 = shift;
my $date3 = shift;
my $time3 = shift;
if (&CompareDates($date1, $time1, $date2, $time2) == -1) {
return 0;
}
if (&CompareDates($date2, $time2, $date3, $time3) == -1) {
return 0;
}
return 1;
}

sub EpochSeconds { #calculates number of seconds since the epoch for the given date/time
my $date = shift;
my $time = shift;
my ($year, $month, $day) = &DateToymd($date);
my ($hour, $minute, $second) = &TimeTohms($time);
my ($d, $h, $m, $s) = &Delta_DHMS(1970, 1, 1, 0, 0, 0,
$year, $month, $day, $hour, $minute, $second);
my $seconds = $s + (60 * $m) + (3600 * $h) + (86400 * $d);
return $seconds;
}

sub SecondsToDateTime { #converts seconds since epoch to date/time
my $seconds = shift;
my $days = int($seconds / 86400);
$seconds -= $days * 86400;
my $hours = int($seconds / 3600);
$seconds -= $hours * 3600;
my $minutes = int($seconds / 60);
$seconds -= $minutes * 60;
my ($year, $month, $day, $hour, $minute, $second) =
&Add_Delta_DHMS(1970, 1, 1, 0, 0, 0, $days, $hours, $minutes, $seconds);
$month = sprintf("%02d", $month);
$day = sprintf("%02d", $day);
$hour = sprintf("%02d", $hour);
$minute = sprintf("%02d", $minute);
$second = sprintf("%02d", $second);
return "$year-$month-$day", "$hour:$minute:$second";
}

sub DateToDayName {
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
my $name = &Day_of_Week_to_Text(&Day_of_Week($year, $month, $day));
$name =~ /^[A-Za-z]{3}/;
$name = $&;
return $name;
}

sub ValiDate {
return @_;

}

sub CheckBusinessDay { # checks to see if date is business day. 1=yes, 0=no
my $date = shift;
my ($year, $month, $day) = &DateToymd($date);
if (Day_of_Week($year,$month,$day) < 6) { return 1; }
else { return 0; }
}

1; #don't remove this line
</pre>

chmod 0700 /usr/home/flowbin/queue.pl

Setup cronjob:
<pre>crontab -e
#move flow data into the queue
1,16,31,46 * * * * /usr/home/flowbin/queue.pl</pre>

==== flow processing: i2b ====
<pre>cat > /usr/home/flowbin/processflows-sql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";
my $sqldir = "/usr/home/sql";
my $sqldirworking = "/usr/home/sql/tmp";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $outfile = "$1-$2:$3.sql";
unless (open (SQL, "+> $sqldirworking/$outfile")) { die "cant open $sqldirworking/$outfile"; }
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 && ($src_ip[2] == 229 || $src_ip[2] == 231)) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
#if ($src_ip[2] == 231) { $d[11] /= 2; $d[10] /= 2; }
}
# note- this is where we filter out IPs only found at i2b
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && ($dst_ip[2] == 229 || $dst_ip[2] == 231)) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] == 229 || $ip[2] == 231) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
#print SQL "$date $ip $ddir $octets\n";
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
print SQL "$sql;\n";

# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}

# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
print SQL "$sql;\n";
}
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
close(SQL);
`bzip2 $sqldirworking/$outfile`;
`mv $sqldirworking/$outfile.bz2 $sqldir/`;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

This script sends the sql files to the traffic server for processing:
<pre>cat > /usr/home/flowbin/sendsql.pl
#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $remote = "69.55.233.199";
my $sqldir = "/usr/home/sql";
my $archive = "/usr/home/archive";
my $sqldirremote = "/data/bwdb2/pending/";
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /bz2$/;

my $r = `scp -Cq $sqldir/$file $remote:$sqldirremote 2>&1`;
# print "scp $sqldir/$file $remote:$sqldirremote";
unless ($?==0) {
push @err, "scp -Cq $sqldir/$file $remote:$sqldirremote ($r)";
}
else {
`mv $sqldir/$file $archive`;
`ssh $remote mv $sqldirremote/$file $sqldirremote/${file}.done`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb2: sendsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'support@johncompanies.com,dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl
#move sql commands to traffic db
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl
</pre>

==== flow processing: castle ====
<pre>
cat > /usr/home/flowbin/processflows.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $log = '/usr/home/flowbin/discards.log';

use Data::Dumper;

BEGIN {
push @INC, "/usr/home/flowbin";
}

use db;

#my $queuedir = "/usr/home/queue";
my $queuedir = "/usr/home/working";
my $archivedir = "/usr/home/archive";

unless ($dry) {
if (-e "$queuedir/.lock") {
open(FILE, "$queuedir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $queuedir/.lock`;
}
}
open(FILE, "> $queuedir/.lock");
print FILE "$$\n";
close(FILE);
}

my $db = db->new();
$db->connect('traffic', '', 'root', '5over3') || die $db->{'error'};

opendir(DIR, $queuedir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
unless($file =~ /^\./) {
$file =~ /([0-9]{4}-[0-9]{2}-[0-9]{2})\.([0-9]{2})([0-9]{2})([0-9]{2})/;
my $date = "$1 $2:$3:$4";
my $condensedDate = $1;
$condensedDate =~ s/-//g;
my $iptotal = {};
my $protototal = {};
my $porttotal = {};

&debug("started file $file at ");
&debug(`date`);
&debug("getting raw flow data (flow-print)");
`cat $queuedir/$file | /usr/local/bin/flow-print -f 5 > /usr/home/working/tmp-$file`;
&debug("aggregating data at ");
&debug(`date`);
unless (open(DATA, "/usr/home/working/tmp-$file")) { die "can't open: $!"; }
LOOP: while (my $line = readline DATA) {
my @d = split /[\s]+/, $line;
if ($d[0] ne '' && $d[0] ne 'Start') {
my $addr = 0;
my $port = 0;

#Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP P Fl Pkts Octets
#0 1 2 3 4 5 6 7 8 9 10 11
#|
# outbound = 2, inbound = 1

my (@src_ip) = split '\.', $d[3];
my (@dst_ip) = split '\.', $d[6];

if ($src_ip[0] == 69 && $src_ip[1] == 55 &&
$src_ip[2] >= 224 && $src_ip[2] <= 239 &&
$src_ip[2] != 229 && $src_ip[2] != 231) { # for castle
# if ($src_ip[0] == 69 && $src_ip[1] == 55 && $src_ip[2] == 229) { # for i2b
$d[2] = 2;
# hack for outbound bulk traffic counted 2x
if ($dst_ip[2] == 234) { $d[11] /= 2; $d[10] /= 2; }
}
elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 &&
$dst_ip[2] >= 224 && $dst_ip[2] <= 239 &&
$dst_ip[2] != 229 && $dst_ip[2] != 231) { # for castle
# elsif ($dst_ip[0] == 69 && $dst_ip[1] == 55 && $dst_ip[2] == 229) { # for i2b
$d[2] = 1;
}
else { next LOOP; }

if ($d[2] == 2) {
$addr = $d[3];
# if the dst-port is low, store that
if ($d[7] <= 1024) { $port = $d[7]; }
# if the src-port is low, store that
elsif ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
}
elsif ($d[2] == 1) {
$addr = $d[6];
# if the dst-port is high, assume its return traffic, try to store src-port if low
if ($d[7] > 1024) {
if ($d[4] <= 1024) { $port = $d[4]; }
else { $port = 99999; }
} else {
$port = $d[7];
}
} else {
next LOOP;
}

my (@ip) = split '\.', $addr;
unless ($ip[0] == 69) { next LOOP; }
unless ($ip[1] == 55) { next LOOP; }
unless ($ip[2] >= 224 && $ip[2] <= 239 && $ip[2] != 229 && $ip[2] != 231) { next LOOP; }
# unless ($ip[2] == 229) { next LOOP; }

my $classC = "$ip[0]_$ip[1]_$ip[2]";

# IP dir
# if ($d[10] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11]; }
#
# if ($d[10] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'pktTotal'} += $d[10]; }
# if ($d[11] < 2147483647) { $porttotal->{$classC}->{$addr}->{$d[2]}->{$port}->{'octetTotal'} += $d[11]; }
$iptotal->{$classC}->{$addr}->{$d[2]}->{'pktTotal'} += $d[10];
$iptotal->{$classC}->{$addr}->{$d[2]}->{'octetTotal'} += $d[11];

$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'pktTotal'} += $d[10];
$protototal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{'octetTotal'} += $d[11];

$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'pktTotal'} += $d[10];
$porttotal->{$classC}->{$addr}->{$d[2]}->{$d[8]}->{$port}->{'octetTotal'} += $d[11];
}
}
close(DATA);
`rm /usr/home/working/tmp-$file`;
&debug("processing ip totals at ");
&debug(`date`);
foreach my $classC (keys(%{$iptotal})) {
$db->query("lock tables dailyIpTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$iptotal->{$classC}})) {
foreach my $dir (keys(%{$iptotal->{$classC}->{$ip}})) {
my $octets = $iptotal->{$classC}->{$ip}->{$dir}->{'octetTotal'};
my $packets = $iptotal->{$classC}->{$ip}->{$dir}->{'pktTotal'};
# $packets = $packets > 2147483647 ? 0 : $packets;
if ($octets > 2147483647) {
my $ddir = $dir==1 ? 'in' : 'out';
`echo "$date $ip $ddir $octets\n" >> $log`;
# $octets = 0;
}
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
my $id = "$ip-$condensedDate-$dir";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $octets, $packets)";
my $sql = "insert into dailyIpTotals_$classC values ('$id', '$date', '$ip', $dir, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into ipTotals values ('$date', '$ip', $dir, $octets, $packets)");
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables ipTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into ipTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
}

sleep 20;
# &debug("processing protocol totals at ");
# &debug(`date`);
# foreach my $classC (keys(%{$protototal})) {
# $db->query("lock tables dailyProtoTotals_$classC write") unless $dry;
# my @values;
# foreach my $ip (keys(%{$protototal->{$classC}})) {
# foreach my $dir (keys(%{$protototal->{$classC}->{$ip}})) {
# foreach my $proto (keys(%{$protototal->{$classC}->{$ip}->{$dir}})) {
# my $octets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'octetTotal'};
# my $packets = $protototal->{$classC}->{$ip}->{$dir}->{$proto}->{'pktTotal'};
# # $octets = $octets > 2147483647 ? 0 : $octets;
# # $packets = $packets > 2147483647 ? 0 : $packets;
# # dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# my $id = "$ip-$condensedDate-$dir-$proto";
# $id =~ s/\.//g;
# push @values, "('$date', '$ip', $dir, $proto, $octets, $packets)";
# my $sql = "insert into dailyProtoTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# # $db->query("insert into protoTotals values ('$date', '$ip', $dir, $proto, $octets, $packets)");
# }
# }
# }
# $db->query("unlock tables") unless $dry;
# my $sql = "insert into protoTotals_$classC values ";
# $sql .= join ',', @values;
# $db->query("lock tables protoTotals_$classC write") unless $dry;
# print "$sql\n" if $dry;
# $db->query($sql) unless $dry;
# $db->query("unlock tables") unless $dry;
# }

&debug("processing port totals at ");
&debug(`date`);
foreach my $classC (keys(%{$porttotal})) {
$db->query("lock tables dailyPortTotals_$classC write") unless $dry;
my @values;
foreach my $ip (keys(%{$porttotal->{$classC}})) {
foreach my $dir (keys(%{$porttotal->{$classC}->{$ip}})) {
foreach my $proto (keys(%{$porttotal->{$classC}->{$ip}->{$dir}})) {
foreach my $port (keys(%{$porttotal->{$classC}->{$ip}->{$dir}->{$proto}})) {
my $octets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'octetTotal'};
my $packets = $porttotal->{$classC}->{$ip}->{$dir}->{$proto}->{$port}->{'pktTotal'};
# $octets = $octets > 2147483647 ? 0 : $octets;
# $packets = $packets > 2147483647 ? 0 : $packets;

# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-protocol-port
my $id = "$ip-$condensedDate-$dir-$proto-$port";
$id =~ s/\.//g;
push @values, "('$date', '$ip', $dir, $proto, $port, $octets, $packets)";
my $sql = "insert into dailyPortTotals_$classC values ('$id', '$date', '$ip', $dir, $proto, $port, $octets, $packets) ON DUPLICATE KEY UPDATE octets=octets+$octets, packets=packets+$packets";
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
# $db->query("insert into portTotals values ('$date', '$ip', $dir, $port, $octets, $packets)");
}
}
}
}
$db->query("unlock tables") unless $dry;

$db->query("lock tables portTotals_$classC write") unless $dry;
# break inserts into 100 records at a time
&debug("inserting $#values +1 values");
while ($#values > 0) {
my $sql = "insert into portTotals_$classC values ";
my $max_index = $#values > 100 ? 100 : $#values;
for (my $i=0; $i<=$max_index; $i++) {
$sql .= shift @values;
$sql .= ',';
}
chop $sql;
print "$sql\n" if $dry;
$db->query($sql) unless $dry;
}
$db->query("unlock tables") unless $dry;
sleep 10;
}

# 12 1 8 1 1= 23
# dailyIpTotals.id = ip(no .'s)-yyyymmdd-direction
# 12 1 8 1 1 3=26
# dailyProtoTotals.id = ip(no .'s)-yyyymmdd-direction-proto
# 12 1 8 1 1 5=28
# dailyPortTotals.id = ip(no .'s)-yyyymmdd-direction-port
#print "finished at ";
#print `date`;
`mv $queuedir/$file $archivedir` unless $dry;
}
}
`rm $queuedir/.lock` unless $dry;

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}

# var full during ft-v05.2005-03-28.084500-0800 and
# 2005-02-24 69.55.226

# all port/daily totals before 2005-04-07
</pre>

<pre>
cat > /usr/home/flowbin/db.pm

#!/usr/bin/perl
#
# $Header: /usr/cvs/newgw/lib/db.pm,v 1.4 2003/06/05 18:20:01 glenn Exp $
#
# Copyright (c) 2003
# e-Monitoring Networks, Inc. All rights reserved.
#
#
package db;

use strict;
use DBI;

sub new {
my $class = shift;
my $self = {};

$self->{'debug'} = 0;
bless $self, $class;

return $self;
}

sub connect {
my $self = shift;
my $dbname = shift;
my $dbhost = shift;
my $dbuser = shift;
my $dbpass = shift;

my $host = '';
if (defined($dbhost)) {
$host = ";host=$dbhost";
}

eval {
$self->debug("connecting to: DBI:mysql:database=$dbname;$host", 1);
$self->{'dbh'} = DBI->connect("DBI:mysql:database=$dbname;$host", $dbuser, $dbpass);
};
if ($self->{'dbh'}) {
return 1;
}
$self->{'error'} = "Error connecting to database $@";
$self->debug("Error connecting to database $@");
return 0;
}

sub query {
my $self = shift;
my $query = shift;

$self->debug($query, 1);
my $sth;
eval {
$sth = $self->{'dbh'}->prepare($query);
};
unless ($sth) {
$self->{'error'} = "error preparing query $@";
$self->debug("error preparing query $@");
return undef;
}
my $qty;
eval {
$qty = $sth->execute;
};
unless ($qty) {
$self->{'error'} = "error executing query $@";
warn "error executing query $@ $query";
return undef;
}
$self->debug("returning $qty, $sth from query", 6);
return ($qty, $sth);
}

sub disconnect {
my $self = shift;

$self->{'dbh'}->disconnect;
return 0;
}

sub debug {
my $self = shift;
my $msg = shift;
my $level = shift || 0;

if ($level < $self->{'debug'}) {
print "$msg\n";
}
return 0;
}
1;
</pre>

mkdir /usr/home/archive
mkdir -p /usr/home/sql/tmp

<pre>crontab -e
#process flows
2,17,32,47 * * * * /usr/home/flowbin/processflows.pl</pre>

==== setup traffic db ====
* Install mysql:
<pre>cd /usr/ports/databases/mysql50-server
make install clean</pre>

cat >> /etc/rc.conf
mysql_enable="YES"

Move db data dir:
/usr/local/etc/rc.d/mysql-server stop
mkdir /usr/home/database/
mv /var/db/mysql/* /usr/home/database/
chown -R mysql:mysql /usr/home/database

Edit database location in startup script:
vi /usr/local/etc/rc.d/mysql-server
# : ${mysql_dbdir="/var/db/mysql"}
: ${mysql_dbdir="/usr/home/database"}

/usr/local/etc/rc.d/mysql-server start

* Install mysql perl database modules:
<pre>
cd /usr/ports/databases/p5-DBI
make install clean
cd /usr/ports/databases/p5-DBD-mysql50
make install clean
(no to SSL support)
</pre>

* Setting up database
<pre>
rehash
/usr/local/etc/rc.d/mysql-server start
mysql -u root
create database traffic;
grant all on *.* to root@localhost identified by '5over3';
grant all on traffic.* to jc@10.1.4.5 identified by '2gMKY3Wt';

</pre>

If this was a new server we'd setup new tables. See [[#mysql_2|mysql]] for how those tables would be setup.

We are assuming here we are moving data from an existing db, here's how that's done (from the current traffic db):
rsync -av --progress /usr/home/database/traffic/ 10.1.4.203:/usr/home/database/traffic/

When you're ready to do the cutover, shut down mysql on both hosts and do one last sync.

==== process flows from bwdb2 ====
On traffic database server (bwdb):

<pre>crontab -e
#import sql from bwdb2
10,25,40,55 * * * * /usr/home/flowbin/processsql.pl</pre>

Add access to mysql:
<pre>mysql -u root -p
grant all on traffic.* to bwdb2@localhost identified by 's1lver4d';
</pre>

<pre>cat > /usr/home/flowbin/processsql.pl

#!/usr/bin/perl

#use strict;
#$debug=1;
#$dry=1;

my $sqldir = "/usr/home/bwdb2/pending";
my $mysql = '/usr/local/bin/mysql';
my @err;
unless ($dry) {
if (-e "$sqldir/.lock") {
open(FILE, "$sqldir/.lock");
my $pid = <FILE>;
chomp($pid);
close(FILE);
if (kill(0, $pid)) {
#another process is using the queue, bail out
exit(0);
}
else {
#dead lock file, remove it
`rm $sqldir/.lock`;
}
}
open(FILE, "> $sqldir/.lock");
print FILE "$$\n";
close(FILE);
}

opendir(DIR, $sqldir);
my @files = readdir(DIR);
closedir(DIR);

foreach my $file (sort @files) {
next unless $file =~ /done$/;
my $r = `bzcat $sqldir/$file | $mysql -u bwdb2 -ps1lver4d traffic`;
unless ($?==0) {
push @err, "bzcat $sqldir/$file | $mysql -u bwdb2 -pxxxxx traffic ($r)";
}
else {
`rm $sqldir/$file`;
}
}

`rm $sqldir/.lock` unless $dry;

if (@err) {
email_support('bwdb: processsql.pl error',join "\n", @err);
}

sub email_support {
my $subj=shift;
my $body=shift;
use Mail::Sendmail;

# prepare message
my %mail = (
To => 'dave@johncompanies.com',
From => 'support@johncompanies.com',
Subject => $subj,
Message => $body,
smtp => 'mail.johncompanies.com',
);
sendmail(%mail) || warn "Error: $Mail::Sendmail::error";
}

sub debug {
my $message = shift;
if ($debug) {
print "$message\n";
}
}
</pre>

chmod 0700 /usr/home/flowbin/processsql.pl

Make sure bwdb is reachable from the outside only to bwdb2:

On nat, add to <tt>/etc/ipnat.rules</tt>
<pre># bwdb
bimap fxp0 10.1.4.203/32 -> 69.55.233.199/32</pre>

Reload:
ipnat -C -F -f /etc/ipnat.rules

Setup firewall rule on firewall:
ipfw add 00094 allow ip from 66.181.18.5 to 69.55.233.199 22
ipfw add 00094 deny ip from any to 69.55.233.199

Setup firewall on bwdb to restrict access now that it's nat'd:
<pre>
cat >> /usr/local/etc/rc.d/boot.sh
ipfw add 1 allow tcp from any to any established
ipfw add 2 allow ip from 10.1.4.0/24,66.181.18.5,69.55.233.195 to me 22
ipfw add 3 allow ip from 10.1.4.5 to me 3306
ipfw add 4 allow ip from 69.55.225.225 53 to me
ipfw add 5 allow ip from 69.55.230.2 25 to me
ipfw add 6 allow ip from me to me 4444
ipfw add 7 allow icmp from any to me
ipfw add 8 allow udp from 10.1.4.203 to 10.1.4.203 dst-port 4444
ipfw add 9 allow udp from 10.1.4.5 to me 161
ipfw add 100 deny ip from any to me
</pre>

chmod 0700 /usr/local/etc/rc.d/boot.sh

From bwdb2, add ssh key:
cat /root/.ssh/id_dsa.pub | ssh 69.55.233.199 'cat - >> /root/.ssh/authorized_keys'

Confirm no password access:
ssh 69.55.233.199 hostname

= bwdb2 =
== Summary ==
This machine tracks and stores network traffic (netflow) at i2b. It is our means to monitor customer bandwidth usage.

* Location: i2b, cab6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.4 There are 2 onboard nic's, one of which is the "listener"
* Hardware: Custom 2U. Single power supply.
* Drives: two 150 GB (2 x 150GB) RAID1 arrays running on a 3ware 7006 RAID card.

== Services Provided ==
* netflow
* bigbrother

== netflow ==

The main function of this server is to run netflow on an eth device in promiscuous mode so as to hear everything happening on the port (wherein all network traffic is mirrored to that port via the cisco swith). Every 15min, it creates a flow file under <tt>/usr/home/flows/</tt> (organized by date). The flow file contains all traffic data for a 15min increment of time.

A cronjob moves that flow file (or files if there are multiple due to some delay)
1,16,31,46 * * * * /usr/home/flowbin/queue.pl

into a processing queue:
<tt>/usr/home/working</tt>

Then a separate file processes whatever flow files it finds there, and builds sql files ready for insertion into the traffic database:
2,17,32,47 * * * * /usr/home/flowbin/processflows-sql.pl

Then yet another process copies the sql files to the traffic database server for processing and insertion into the mysql database:
8,23,38,53 * * * * /usr/home/flowbin/sendsql.pl

== Regular maintenance ==
*[[Routine_Maintenance#Adaptec_Controllers|Check RAID array]]

* if space becomes tight, move sql files and flow files to backup server, both located in <tt>/usr/home/flowbin/archive</tt>

= firewall (newgateway) =

== Summary ==

This machine is the primary (only) firewall for the entire network at castle.

* Location: castle, cab 3-8
* OS: FreeBSD 4.11 x86
* Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
* Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* snmp
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

<pre>hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"

#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248</pre>

== Cronjobs ==
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside <tt>/etc/daily.local</tt> you will see a call to <tt>/etc/makepiperules.pl</tt>
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
# enter a rule that allows all UDP to go through
# send an emergency email to support and indicating an attack is in progress
# send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
# wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl
To run:
cd /usr/home/user
./doswatch.pl &

To kill;
fg
^C

It writes its findings to /usr/home/user/doswatch.log

= backup1 =

== Summary ==

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

* Location: castle, cab 3-8
* OS: Ubuntu 8.04.1 server x86
* Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
* Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

== Services provided ==
* backup via rsync
* mysql
* nfs
* snmp
* bigbrother

== Usage and Notes ==
* all data is stored under /data
* virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: <tt>/backup1</tt>), and they all have their ssh keys setup to allow passwordless rsync's
* each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
* at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
* cancelled customer systems are compressed and stored under <tt>/data/deprecated</tt>
* archived bwdb2 flow files are stored under <tt>/data/bwdb2</tt>
* critical files from backup2 are stored under <tt>/data/backup2</tt>

== Cronjobs ==
<pre>
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log</pre>
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. It will email support@johncompanies.com at it’s conclusion. This email can be deleted, however note when it begins to take significantly longer to complete, ie runs past 2200 pm – this usually indicates a problem on the backup server.

<pre>10,25,40,55 * * * * /usr/local/sbin/processsql.pl
</pre>
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
<pre>0 0 * * * /usr/local/sbin/3wraidchk
</pre>
checks the health of the RAID array

== Regular maintenance ==
*[[Routine_Maintenance#Free_up_space_on_backup1|Remove old backups]]
*[[Routine_Maintenance#3ware|Check on auto-verify]]

== build ==

<pre>Setup raid5 with a boot vol of 12G 5.45tb
12G boot
4664 GB

Install ubuntu 8.04

Swap 4G

Don’t format data drive

http://www.unixgods.org/~tilo/linux_larger_2TB.html

parted /dev/sdb
print
mklabel gpt
print

#Disk /dev/sdb: 4987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags

mkpart primary ext3 0 4987GB
print

#Disk /dev/sdb: 5987GB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt

#Number Start End Size File system Name Flags
# 1 17.4kB 4987GB 4987GB primary

quit

mkfs.ext3 /dev/sdb1
#mke2fs 1.40.8 (13-Mar-2008)
#Filesystem label=
#OS type: Linux
#Block size=4096 (log=2)
#Fragment size=4096 (log=2)
#304390144 inodes, 1217544183 blocks
#60877209 blocks (5.00%) reserved for the super user
#First data block=0
#Maximum filesystem blocks=0
#37157 block groups
#32768 blocks per group, 32768 fragments per group
#8192 inodes per group
#Superblock backups stored on blocks:
# 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
# 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
# 102400000, 214990848, 512000000, 550731776, 644972544
#
#Writing inode tables: 967/37157

mkdir /data

#root@backup1:~# df -h
#Filesystem Size Used Avail Use% Mounted on
#/dev/sda2 8.3G 540M 7.3G 7% /
#varrun 1013M 40K 1013M 1% /var/run
#varlock 1013M 0 1013M 0% /var/lock
#udev 1013M 56K 1013M 1% /dev
#devshm 1013M 0 1013M 0% /dev/shm
#/dev/sdb1 4.5T 192M 4.3T 1% /data

apt-get update
apt-get upgrade
apt-get install snmp snmpd ntp nfs-kernel-server

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

vi /etc/ntp.conf
server 10.1.4.5

scp root@10.1.4.3:/root/.ssh/authorized_keys /root/.ssh/
cd /root/
ssh-keygen -t dsa
echo "10.1.4.3 backup2" >> /etc/hosts

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'

ssh backup2

vi /root/.bashrc
export PS1="[\u@\h \w]# "
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

source /root/.bashrc

echo "# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on runlevel 0
stop on runlevel 1
stop on runlevel 6

respawn
exec /sbin/getty 38400 ttyS0" > /etc/event.d/ttyS0

vi /boot/grub/menu.lst

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

append to kernel lines:
console=tty0 console=ttyS0,38400n8

show menu:
#hiddenmenu

echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

# to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd

echo "bb:x:1984:" >> /etc/group

pwconv

mkdir /home/bb
chown bb.bb /home/bb

cd ~bb
scp backup2:/mnt/data4/build/bb/bb-linux.tar .

tar xf bb-linux.tar

cd /home/bb/bbc1.9e-btf/etc

echo "10.1.4.5 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.4.8 backup1.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx)

chmod +r /var/log/messages

./bbchkcfg.sh
#(y to questions)
./bbchkhosts.sh
#(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src

#make; make install
cd ..
./runbb.sh start
more BBOUT
(look for errors)
exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

echo '/data 10.1.4.0/24(rw, no_root_squash,async,no_subtree_check)' >> /etc/exports

/etc/init.d/nfs-kernel-server restart

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

echo '10.1.4.8 backup1' >> /etc/hosts
echo '/dev/sdb1 /data ext3 rw,noatime 0 0' >> /etc/fstab

to install digi drivers:

wget http://ftp1.digi.com/support/driver/40002086_n.tgz
apt-get install linux-image-2.6.24-19-server
apt-get install linux-source-2.6.24 (not needed?)
apt-get install linux-headers-2.6.24-19-server
apt-get install make
apt-get install gcc
apt-get install g++
apt-get install libncurses5-dev
apt-get install expect
apt-get install libdbi-perl libdate-calc-perl libdbd-mysql-perl

cd /usr/src; ln -s linux-headers-2.6.24-19-server linux
./configure
make all
make install
make postinstall

/usr/bin/dgrp_cfg_node -v -v init el 65.116.11.2 8

apt-get install mysql

mkdir /data/mysql
chown mysql:mysql /data/mysql
/etc/init.d/mysql stop
mv /var/lib/mysql/* /data/mysql/
mv /data/mysql/ib_* /var/lib/mysql/
vi /etc/mysql/my.cnf
(change datadir to /data/mysql)
vi /etc/apparmor.d/usr.sbin.mysqld
add:
/data/mysql/ r,
/data/mysql/** rwk,
Comment out:
# /var/lib/mysql/ r,
# /var/lib/mysql/** rwk,

/etc/init.d/apparmor restart
/etc/init.d/mysql start

tw_cli /c0/u0 set ignoreECC=on
tw_cli /c0/u0 set storsave=balance
tw_cli /c0/u0 set cache=on

0 0 * * * /usr/local/sbin/3wraidchk
</pre>

= backup2 =

== Summary ==

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

* Location: castle, cab 3-7
* OS: FreeBSD 6.1 x86
* Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
* Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
* Drives:
**3ware 7500-8:
***200 GB JBOD (1 x 200G) labeled 0-0
***500 GB RAID5 (3 x 250G) 0-1 thru 0-3
***700 GB RAID5 (4 x 250G) 0-4 thru 0-7
**3ware 7500-8:
***700 GB RAID5 (4 x 250G) 1-0 thru 1-3
***700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

In case of an outage, nfs will hang on all connected servers until the nfs service returns. If you can't get backup2 back online, you can get nfs running elsewhere and fake backup2's MAC's: priv: 00:0e:0c:59:c1:a6, pub: 00:07:e9:5b:c6:45

To configure:
ifconfig fxp0 link 00:90:27:f9:0a:d9

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* snmp
* bigbrother

== Usage ==
* all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: <tt>/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4</tt>
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/mnt/data2/iso</tt>
* this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
* this server serves as an archive for exported db data from bwdb and old flow files.
* isys backs up here
* customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
* in <tt>/mnt/data4</tt> there are lots of useful things used for building our vps servers, customer servers, and management scripts:
** <tt>/bin</tt>: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
** <tt>/build</tt>: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
** <tt>/vzrpms</tt>: contains the OS templates for many-to-most of the OS's we offer on vz systems

== Cronjobs ==
* backs itself up nightly to nfs-mounted backup1 (mountpoint: <tt>/backup2</tt>)

== Regular maintenance ==
*[[Routine_Maintenance#3ware|Check on health]]

= backup3 =
== Summary ==
This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

* Location: i2b, cab 6
* OS: Ubuntu 10.04.1 server amd64
* Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
* Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

== Services provided ==
* backup via rsync and nfs
* samba
* nfs
* digi realport
* snmp
* bigbrother

== Usage ==
* all data is stored under /data
* iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under <tt>/data/iso</tt>
* this server serves as an archive for exported db data from bwdb and old flow files.
* inftrastructure machines at i2b back up here
* customers are nfs-moutned under /data/customers as file-backed loopback devices

== management scripts ==
* mkbackups

== Cronjobs ==
0 0 * * * /usr/local/sbin/arecaraidchk
RAID checks

35 4 * * * /usr/local/sbin/snapshot_archive
Rotate daily snapshots for infrastructure machine backups

== Regular maintenance ==
*[[Routine_Maintenance#Areca|Check on RAID health]]

== Build ==

=== BIOS Config ===
disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

=== Install OS ===
<pre>Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba</pre>

=== DNS and private IP ===

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net
<pre>vi /etc/network/interfaces

auto eth0
iface eth0 inet static
address 69.55.229.4
netmask 255.255.255.0
network 69.55.229.0
broadcast 69.55.229.255
gateway 69.55.229.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 69.55.229.3 66.181.0.2
dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
address 69.55.231.2
netmask 255.255.255.0
network 69.55.231.0
broadcast 69.55.231.255

auto eth1
iface eth1 inet static
address 10.1.2.3
netmask 255.255.255.0
network 10.1.2.0
broadcast 10.1.2.255

</pre>

=== Install packages ===
<pre>apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server</pre>

=== tweak grub, enable serial ===

<pre>vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub</pre>

<pre>echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf</pre>

=== install realport (digi) driver ===

give the digi an ip with DgIpServ.exe

<pre>cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults</pre>

configure ports:
dgrp_cfg_node init el 10.1.2.10 16

try connecting with:
cu -l /dev/ttyel00 -s 38400

=== shell, ntp, ssh key, hosts ===

Shell autocompletion search:
<pre>echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc</pre>

Setup ntp:
<pre>vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com</pre>

Generate ssh keys:
<pre>cd /root/
ssh-keygen -t dsa</pre>
Defaults, no password

Setup hosts:
<pre>echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts</pre>

Copy keys to servers where we need passwordless login:
<pre>cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'</pre>

Setup shell:
<pre>vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity jcread 10.1.4.5
rocommunity jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
/etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
echo "bb:x:1984:" >> /etc/group
pwconv

Create home:
mkdir /home/bb
chown bb.bb /home/bb
cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc</pre>

Configure main bb server:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts:
<pre>echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab</pre>

vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh
(remove all | SORT xxxx since SORT is broken)

chmod +r /var/log/messages

./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
<pre>cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src</pre>

<pre>make; make install
cd ..
./runbb.sh start
more BBOUT</pre>
(look for errors)
exit

<pre>vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"
</pre>
(before the exit 0)

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule:
ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

<pre>vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199</pre>

=== raid check ===

==== 3ware ====
<pre>
scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk</pre>

==== areca ====
<pre>
cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info
</pre>

<pre>scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk
</pre>

<pre>cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1
</pre>

=== misc binaries ===

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive
(remove entries)

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host:

<pre>
cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];

my %mail = (
To => $to,
From => $to,
Subject => $sub,
Message => '',
smtp => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";

</pre>

chmod 0700 /bin/mail

=== mkbackup ===

mkdir /data/customers

<pre>cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
cid=$1
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $2; then
gb=$2
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test $3; then
ip=$3
else
echo "ERROR: Usage: mkbackup cid GB ip Terminating."
exit
fi

if test -e /data/customers/${cid}-file; then
echo "ERROR: /data/customers/${cid}-file exists"
exit
else
echo "touch /data/customers/${cid}-file"
touch /data/customers/${cid}-file
count=`echo $gb|awk '{print $1*1000}'`
echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
echo "ERROR: /data/customers/$cid exists"
exit
else
echo "mkdir /data/customers/${cid}"
mkdir /data/customers/${cid}
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
mount -o loop /data/customers/${cid}-file /data/customers/$cid
df -h /data/customers/$cid

echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
echo "" >> /etc/nfs_backup_mounts.sh

echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
/etc/init.d/nfs-kernel-server restart
tail /var/log/messages
fi</pre>

chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local
add:
/etc/nfs_backup_mounts.sh

=== samba ===

apt-get install samba

vi /etc/samba/smb.conf

; comment out any mounts, add:

<pre>[data]
read only = yes
locking = no
path = /data/iso
guest ok = yes</pre>

/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2

<pre>cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .</pre>

=== Moving from one server to another ===

Here are the steps you would take to move settings and data from one server to a new backup server:

* rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/
after umounting all the customers, copy over the (empty) directories separately:
for f in `find . -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done

* copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh

* copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local

* copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports

* edit /etc/hostname on both machines (set current to oldbackup3)

* edit /etc/network/interfaces (swap IPs).

* stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh

* reboot both servers @ same time

* check everything out

* run /etc/nfs_backup_mounts.sh on new server

* if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

= console =

== Summary ==
This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

* Location: castle, cab 3-8
* OS: SunOS 5.8 (solaris)
* Networking: Priv IP: 10.1.4.4
* Hardware: Sun Netra

To connect to consoles, ssh in as user 'console' and use the <tt>tip</tt> command to connect to devices listed in <tt>/etc/remote</tt>

i.e.
tip switch-p1
tip jail1

== Configuring digi/ports ==

=== /etc/remote ===
This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to:

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. <tt>jail4</tt>) and the speed (i.e. <tt>9600</tt>). You can decipher which port on the digi each line corresponds to by the <tt>CP001s or CO001s</tt> (port 1 on digi1 and digi2), <tt>CP002s or CO002s</tt> (port 2 on digi1 and digi2)

=== drpadmin ===
The tool you use to configure a device to a digi box is drpadmin:

<pre>bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0 10.1.4.10 32 CO 771 never 1027
1 10.1.4.11 32 CP 771 never 1027
2 65.116.11.2 8 el 771 never 1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :</pre>

Use those commands above to modify the devices available.

== Switching IP/hostname ==

Edit:
/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up

= devweb =

We do web development on devweb.johncompanies.com

Currently this is a jail running on jail17 / 69.55.230.8

If the jail is restarted, you will need to manually restart the web service with:
httpsdctl restart

All website development work should be done here first. It works exactly like and is setup like our [[Management_System_/_Public_Website_/_Signup_/_Account_Manager|main site]].

= firewall2 =

== Summary ==

This machine is the primary (only) firewall for the entire network at i2b.

* Location: i2b, cab 6
* OS: FreeBSD 6.4 x86
* Networking: Priv IP: 10.1.2.2, Pub IPs: 66.181.18.3 (external), 69.55.229.1 & 69.55.231.1 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. TODO: describe NIC location/orientation

* Hardware: 2 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
* Drives: 73 GB (2 x 73GB) RAID1 array running on an LSI MegaRAID SCSI 320 PCI RAID card.

== Services Provided ==
* firewall (ipfw)
* bigbrother

== Firewall Rule Configuration ==

See [[FreeBSD_Reference#Firewall_Rule_Configuration|Firewall Rule Configuration]] for more discussion on how to actually manipulate firewall rules.

== Disaster Recovery ==

TODO: need backup f/w and instructions on how to move cables.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)

TODO

Here's the config on the live firewall:

<pre>kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias0="inet 69.55.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias1="inet 65.50.228.1 netmask 255.255.255.0"
ifconfig_bge1_alias2="inet 65.50.229.1 netmask 255.255.255.0"
ifconfig_bge1_alias3="inet 65.50.230.1 netmask 255.255.255.0"
ifconfig_bge1_alias4="inet 65.50.231.1 netmask 255.255.255.0"
ifconfig_bge1_alias5="inet 65.50.232.1 netmask 255.255.255.0"
ifconfig_bge1_alias6="inet 65.50.233.1 netmask 255.255.255.0"
ifconfig_bge1_alias7="inet 65.50.234.1 netmask 255.255.255.0"
ifconfig_bge1_alias8="inet 65.50.235.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"
</pre>

== Cronjobs ==
30 3 * * * /usr/local/etc/rsync.backup
Backup to backup3

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
Capture counts periodically

0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
This script will create <tt>/etc/firewall.sh</tt> which contains all the firewall and pipe rules in place at the time the script was run.

*/5 * * * * /usr/local/sbin/lsiraidchk
Checking the health of the RAID array

== DOS attacks ==

See [[FreeBSD_Reference#Handling_a_DoS_attack|Handling a DoS attack]] regarding how to handle a DOS attack.

== build ==

<pre>partition map:
/ 58g
swap 4g
/var 512m
/tmp 512m
/usr 5.5g

4. edit /etc/make.conf
echo "WITHOUT_X11=yes \
KERNCONF=firewall2 \
BOOT_COMCONSOLE_SPEED=115200" >> /etc/make.conf

5. add settings to /boot/loader.conf and /boot.config

echo "-Dh" >> /boot.config

echo 'console="comconsole,vidconsole" \
boot_multicons="YES" \
boot_serial="YES" \
comconsole_speed="115200"' >> /boot/loader.conf

6. turn off all ttyv's except 0 and 1 in /etc/ttys
also turn on ttyd0, change type to vt100:
vi /etc/ttys

ttyv2 "/usr/libexec/getty Pc" cons25 off secure
ttyv3 "/usr/libexec/getty Pc" cons25 off secure
ttyv4 "/usr/libexec/getty Pc" cons25 off secure
ttyv5 "/usr/libexec/getty Pc" cons25 off secure
ttyv6 "/usr/libexec/getty Pc" cons25 off secure
ttyv7 "/usr/libexec/getty Pc" cons25 off secure
# Serial terminals
# The 'dialup' keyword identifies dialin lines to login, fingerd etc.
ttyd0 "/usr/libexec/getty std.9600" vt100 on secure

kill -1 1

on console server:
vi /etc/remote
(rename port to jail8 depending on where and which digi plugged into)
test serial console

7. populate hosts
echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

8. put key in authorized_keys on backup3
cd
ssh-keygen -t dsa -b 1024
(default location, leave password blank)

Punch a hole in firewall1 to allow traffic to backup servers @ castle:

ipfw add 99 allow ip from 66.181.18.0/27 to 69.55.230.10 22
ipfw add 95 allow ip from 66.181.18.0/27 to 69.55.230.11 22

cat /root/.ssh/id_dsa.pub | ssh backup3 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys'
cat /root/.ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

confirm that you can ssh to backup3 and backup 2 without getting a login prompt

ssh backup3 hostname

ssh backup2 hostname

ssh backup1 hostname

10. edit root's path and login script:
vi /root/.cshrc

Change alias entries (add G):
alias la ls -aG
alias lf ls -FAG
alias ll ls -lAG
alias ls ls -AG
alias mbm mb mount
alias mbu mb umount

and alter the prompt, set the following:
set prompt = "`/bin/hostname -s` %/# "

11. install cvsup
cd /usr/ports/net/cvsup-without-gui
make install clean; rehash; mail -s 'cvs installed' dave.boodman@vtext.com < /dev/null

12. get latest sources for this release:
cd /usr/src
echo "*default host=cvsup4.freebsd.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
src-all" > sup

cvsup sup ; mail -s 'cvs sup done' dave.boodman@vtext.com < /dev/null

13. configure new kernel.

cd /usr/src/sys/i386/conf
scp backup2:/mnt/data4/build/freebsd/firewall2-6.4 ./firewall2

15. build, install kernel and world

cd /boot

mv kernel kernel.GENERIC
cd kernel.GENERIC
cd /usr/src
make buildkernel installkernel

make buildworld ; mail -s 'buildworld done' dave.boodman@vtext.com < /dev/null
(2450: 1:56min, supermicro: 59mins, 2950: 38mins)
make installworld
(2450: 3min, supermicro: 1min, 2950: :34)
mergemaster -i

17. populate /etc/rc.conf with IPs and NFS settings
vi /etc/rc.conf

kern_securelevel_enable="NO"
portmap_enable="NO"
sendmail_enable="NO"
usbd_enable="YES"
gateway_enable="YES"

xntpd_enable="YES"
nfs_client_enable="YES"
nfs_reserved_port_only="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.1.2.2"

ifconfig_bce1="inet 10.1.2.2 netmask 255.255.255.0"
fsck_y_enable="YES"
background_fsck="NO"

defaultrouter="66.181.18.2"
hostname="firewall2.johncompanies.com"
ifconfig_bge0="inet 66.181.18.3 netmask 255.255.255.224"
ifconfig_bge1="inet 69.55.229.1 netmask 255.255.255.0"
ifconfig_fxp0="inet 10.1.2.2 netmask 255.255.255.0"
sshd_enable="YES"
usbd_enable="YES"

20. reboot. Confirm new kernel is loaded

uname -a

21. update ports:
cd /usr/ports
echo "*default host=cvsup4.FreeBSD.org\
*default base=/usr\
*default prefix=/usr\
*default release=cvs tag=RELENG_6_4\
*default delete use-rel-suffix\
*default compress\
ports-all tag=." > sup

cvsup sup; mail -s 'cvs sup ports done' dave.boodman@vtext.com < /dev/null

22. Install raid mgmt tool

# linux base
cd /usr/ports/devel/libtool22
make install base

cd /usr/ports/emulators/linux_base-fc4
make install clean

#linux-megamgr-5.20
cd /usr/ports/sysutils/linux-megamgr
make install clean

# megarc-1.51
cd /usr/ports/sysutils/megarc
make install clean

Test:
rehash; megarc -ldInfo -a0 -l0

23. install rsync from ports
cd /usr/ports/net/rsync
make install clean

choose default options

25. install bb client
adduser
Username: bb
Full name: bb
Uid (Leave empty for default): 1984
Login group [bb]:
Login group is bb. Invite bb into other groups? []:
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bb]:
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: yes
Lock out the account after creation? [no]:
Username : bb
Password : <random>
Full Name : bb
Uid : 1984
Class :
Groups : bb
Home : /home/bb
Shell : /bin/sh
Locked : no
OK? (yes/no): yes

cd /usr/home/bb
scp backup2:/mnt/data4/build/bb/bb-freebsd.tar .
tar xvf bb-freebsd.tar

edit /home/bb/bbc1.9e-btf/etc/bb-hosts with something like:
echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh \
10.1.2.1 firewall2.johncompanies.com # ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts

vi /home/bb/bbc1.9e-btf/ext/openfiles
MACHINE="firewall2,johncompanies,com" # HAS TO BE IN A,B,C FORM

cd /usr/home/bb/bbc1.9e-btf/etc
./bbchkcfg.sh
(y to questions)
./bbchkhosts.sh
(ignore ssh errors)
cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..

vi /usr/home/bb/bbc1.9e-btf/bin/bb-cpu.sh
$1 $TOPARGS > $BBTMP/TOP.$$
# /usr/local/jail/bin/jtop > $BBTMP/TOP.$$

./runbb.sh start
more BBOUT
(look for errors)
exit

echo 'su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"' > /usr/local/etc/rc.d/bb.sh
chmod +x /usr/local/etc/rc.d/bb.sh

Punch a hole in the firewall to allow it to communicate with bb monitor:

ipfw add 00096 allow ip from 66.181.18.0/27 to 69.55.230.2

27. configure bb on mail:
vi /usr/home/bb/bbsrc/bb1.9e-btf/etc/bb-hosts
66.181.18.3 firewall2.johncompanies.com # ssh

su bb
cd
bbsrc/bb/runbb.sh restart ; exit

29. configure ntp
echo "server 10.1.2.1" > /etc/ntp.conf

/usr/sbin/ntpd -p /var/run/ntpd.pid
sleep 2; ntpq -p
(confirm it’s able to reach our time server)

echo '/usr/sbin/ntpd -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift' > /usr/local/etc/rc.d/ntp.sh
chmod 0700 /usr/local/etc/rc.d/ntp.sh

30. fwd and reverse lookups on ns1c
vr johncompanies.com
(edit the PTR too)

33. setup backups
echo '#\!/bin/sh\
backupdir=/data/firewall2/current\
\
## ENTRY /etc ' > /usr/local/etc/backup.config

on backup3:
setup backup dirs:
ssh backup3 mkdir -p /data/firewall2/current

on backup3, add the system to
vi /usr/local/sbin/snapshot_archive

scp 69.55.230.2:/usr/local/etc/rsync.backup /usr/local/etc/rsync.backup
vi /usr/local/etc/rsync.backup
backup1 > backup3

crontab -e
1 0 * * * /usr/local/etc/rsync.backup

34. mkdir /root/logs

35. edit sshd_config for security
vi /etc/ssh/sshd_config
ListenAddress 66.181.18.3
ListenAddress 10.1.2.1

kill -1 `cat /var/run/sshd.pid`

35. raid chk

cat > /usr/local/sbin/lsiraidchk
#!/usr/bin/perl

my @out = split "\n", `megarc -ldInfo -a0 -Lall|grep Status:`;

foreach (@out) {
if ($_ =~ /DEGRADED/) { $date = `date`; chomp $date; `echo "$date: RAID ARRAY DEGRADED" >> /var/log/messages`; }
#print $_;
}

36. add crontab entries
crontab -e
30 3 * * * /usr/local/etc/rsync.backup
0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3 4 5
59 23 30 * * /sbin/ipfw show > /tmp/ipfw_count
3 0 30 * * /sbin/ipfw show > /tmp/ipfw_count
0 3 * * * /etc/makefwrules.pl; /etc/makepiperules.pl;
*/5 * * * * /usr/local/sbin/lsiraidchk

#10 0 * * * rm /var/spool/clientmqueue/*

scp /etc/makefwrules.pl user@64.163.14.48:~
scp /etc/makepiperules.pl user@64.163.14.48:~
mv /home/user/makefwrules.pl /etc
mv /home/user/makepiperules.pl /etc
touch /etc/firewall.sh
mkdir /etc/oldrules/

other binaries

scp /usr/local/bin/rulemaker user@64.163.14.48:~
mv ~user/rulemaker /usr/local/sbin
scp ~user/Sendmail.pm user@64.163.14.48:~
scp ~user/doswatch.pl user@64.163.14.48:~

Setup basic ruleset

ipfw add 00009 count udp from any to any
ipfw add 00010 allow tcp from any to any established
ipfw add 00012 deny tcp from any to any tcpflags syn tcpoptions !mss
ipfw add 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18
ipfw add 00012 deny tcp from any to any tcpflags syn,fin
ipfw add 00012 deny tcp from any to any tcpflags fin,psh,rst,urg
ipfw add 00012 allow icmp from any to any
ipfw add 00014 deny tcp from any to any dst-port 135
ipfw add 00150 skipto 65535 ip from any to any via em1 in

IPKVM3:
00098 allow ip from { 69.55.230.6 or 69.55.230.7 } to 69.55.230.10 dst-port 139
00098 deny ip from any to 69.55.230.10 dst-port 139
</pre>

= wiki =

The wiki (mediawiki) runs on nat2 in a jail running off 69.55.229.8

The backup wiki lives on virt13 in CT 5 / 69.55.230.18

== Setup jail ==

<pre>
mkdir /mnt/data1/wiki-dir
cd /usr/src
make installworld DESTDIR=/mnt/data1/wiki-dir
cd etc
make distribution DESTDIR=/mnt/data1/wiki-dir

mount -t devfs devfs /mnt/data1/wiki-dir/dev
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset

cd /mnt/data1/wiki-dir

ln -sf dev/null kernel

scp jail9:/usr/local/sbin/jkill /mnt/data1/wiki-dir/sbin

jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh
csh

touch /etc/fstab
echo 'network_interfaces=""\
hostname="wiki.johncompanies.com"\
kern_securelevel_enable="NO"\
sendmail_enable="YES"\
sshd_enable="YES"' > /etc/rc.conf

echo "nameserver 69.55.229.3\
nameserver 69.55.225.225" >> /etc/resolv.conf

vi /etc/crontab

(remove the adjkerntz lines )

vi /etc/periodic/security/100.chksetuid

replace: MP=`mount -t ufs | grep -v " nosuid" | awk '{ print $3 }' | sort`
with: MP='/' (use single quotes)

mkdir -p /usr/compat/linux/dev

adduser

Username: user
Full name: user
Uid (Leave empty for default):
Login group [user]:
Login group is user. Invite user into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/user]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]: y
Lock out the account after creation? [no]:
Username : user
Password : <random>
Full Name : user
Uid : 1001
Class :
Groups : user
Home : /home/user
Home Mode :
Shell : /bin/sh
Locked : no
OK? (yes/no): y
adduser: INFO: Successfully added (user) to the user database.
adduser: INFO: Password for (user) is: 901gmYjO
Add another user? (yes/no): n
Goodbye!

vi /usr/home/user/.profile
TERM=vt100; export TERM

tzsetup

newaliases

rm /sbin/halt /sbin/reboot
ln /sbin/jkill /sbin/halt
ln /sbin/jkill /sbin/reboot

vi /etc/syslog.conf
#*.err;kern.warning;auth.notice;mail.crit /dev/console
*.err;kern.warning;auth.notice;mail.crit /var/log/messages

exit
exit

cd libexec
chflags noschg ld-elf32.so.1
chflags noschg ld-elf.so.1
mv ld-elf32.so.1 ld-elf32.so.1-orig
ln ld-elf.so.1 ld-elf32.so.1
chflags schg ld-elf.so.1
chflags schg ld-elf32.so.1

cp -r /usr/ports /mnt/data1/wiki-dir/usr

cat > /usr/local/etc/rc.d/wiki.sh
mount -t devfs devfs /mnt/data1/wiki-dir/dev/
devfs -m /mnt/data1/wiki-dir/dev rule -s 3 applyset
jail /mnt/data1/wiki-dir wiki.johncompanies.com 69.55.229.8 /bin/sh /etc/rc

chmod 0700 /usr/local/etc/rc.d/wiki.sh
</pre>

== mediawiki setup ==

<pre>
cd /usr/ports/distfiles/
fetch http://downloads.mysql.com/archives/mysql-5.5/mysql-5.5.4-m3.tar.gz
cd /usr/ports/databases/mysql55-server
make install clean

cd /usr/ports/distfiles/
fetch http://downloads.php.net/johannes/php-5.3.2.tar.bz2
cd /usr/ports/lang/php52
make install clean

cd /usr/ports/www/apache22
make install clean

cd /usr/local/www/
fetch http://download.wikimedia.org/mediawiki/1.19/mediawiki-1.19.2.tar.gz
tar xzf mediawiki-1.19.2.tar.gz
mv mediawiki-1.19.2 wiki

vi /usr/local/etc/apache22/httpd.conf

Include etc/apache22/extra/vhost-wiki.conf
Listen 443

cat > /usr/local/etc/apache22/extra/vhost-wiki.conf
<VirtualHost *:443>
ServerAdmin support@johncompanies.com

DocumentRoot /usr/local/www/wiki
# <Directory />
# Options FollowSymLinks
# AllowOverride None
# Order deny,allow
# </Directory>
<Directory /usr/local/www/wiki>
Options Indexes FollowSymLinks MultiViews
Deny from all
AllowOverride AuthConfig
Order allow,deny
#Allow from 69.55.233.195
#Allow from boody.dyndns.org
</Directory>

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

ErrorLog ${APACHE_LOG_DIR}/error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog ${APACHE_LOG_DIR}/access.log combined

Alias /doc/ "/usr/share/doc/"
<Directory "/usr/share/doc/">
Options Indexes MultiViews FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
Allow from 127.0.0.0/255.0.0.0 ::1/128
</Directory>

SSLEngine on
SSLCertificateFile /etc/apache2/ssl/server.crt
SSLCertificateKeyFile /etc/apache2/ssl/server.key

</VirtualHost>

</pre>