Infrastructure Machines: Difference between revisions

From JCWiki
Jump to navigation Jump to search
← Older editNewer edit →
Line 405: Line 405:
* nfs
* nfs
* snmp
* snmp
* bigbrother


== Usage ==
== Usage ==

Revision as of 17:03, 13 December 2012

mail

Summary

This machine is the swiss army knife of the company, playing host to many services and functions.

  • Location: castle, cab 3-8
  • OS: FreeBSD 4.10 x86
  • Networking: Priv IP: 10.1.4.5, Pub IPs: 69.55.230.2, 69.55.225.225 (ns1c jail), 69.55.230.9. 1 onboard and 1 PCI
  • Hardware: Dell 2450. 4 SCSI SCA drive bays (2 columns of 2, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
  • Drives: two 36 GB (2 x 36GB) RAID1 arrays running on an Adaptec-based, Dell-branded (perc) RAID card.

Services Provided

  • mail
  • web
  • mysql
  • bigbrother server/pager
  • snmp

email

This server hosts mail for johncompanies.com (mail.johncompanies.com). Sendmail 8.13.6/8.13.6 is listening on 69.55.230.2 port 25 for incoming mail. Relaying is allowed per /etc/mail/relay-domains Other addresses (aliases) are defined per /etc/mail/aliases

The following active users have mail hosted on this server:

  • dave
  • linux
  • support
  • payments
  • sales
  • tech1
  • info

Traditionally, mail is checked via shell apps (pine). qpopper (pop3s) is running to allow mail downloading. Checking mail in this way causes an opened INBOX in pine to lock read-only. For this reason, we tee incoming mail to support and linux to tech1.

Procmail rules are setup to filter spam and send text messages. They are enabled for info, support, linux, tech1, dave and can be found in ~/Procmail/, for example: 

# more ~support/Procmail/rc.emergency
:0c # use c only if you want to forward a copy and file the original later
* ^Subject:.*\<emergency\>
* ! ^Subject:\<re\>
  {
   :0h
   FROMANDSUBJECT=|formail -XFrom: -XSubject:

   :0fwh
   | /usr/local/bin/formail -I"Subject: " -I"To: pager@johncompanies.com" ; echo $FROMANDSUBJECT ; echo

   :0
  ! -t
  }

control: cd /etc/mail; make stop (stop), cd /etc/mail; make start (start)

web

This server hosts the johncompanies public website, including the ordering system, management system and account manager. Apache/1.3.31 is listening on 69.55.230.9 ports 80 and 443

  • webroot: /usr/local/www/
  • config: /usr/local/etc/apache/
  • logs: /var/log/httpd-error.log /var/log/httpd-access.log
  • control: apachectl stop (stop), apachectl startssl (start)

Web components:

public site

signup

management (mgmt)

account manager (AM)

mrtg

All configuration is done via *.cfg files. The main load graph is found in mrtg1.cfg All other config files are for various switches. Switch config files are rebuilt out of a cron jobs running on mail. This ensures if we change a port name (desc) that the mrtg we look at has the latest info. So if you want to change port naming, please do it in the switch itself. If you have problems getting new devices setup or change existing devices you may need to change permissions on the cfg file as well as the data file in /usr/local/www/mgmt/mrtg/data, including removal of the rrd file if necessary.

mysql

mysql 4.1.22 is running on port 3306

  • datadir: /mnt/data1/db/mysql/
  • config: /etc/my.cnf
  • database: jc
  • control: /usr/local/etc/rc.d/mysql-server.sh stop (stop), /usr/local/etc/rc.d/mysql-server.sh start (start)

bigbrother

There is a client running on mail (which monitors the services running on mail and mail itself), installed under /usr/home/bb/bbc1.9e-btf
And the big brother pager/server (which displays information gathered from all bb-monitored machines, including mail) is installed under /usr/home/bb/bbsrc/bb1.9i-btf

Both are running under the user bb

Refer to BigBrother for more about use.

DNS (ns1c.johncompanies.com)

ns1c is a jail running on the mail server, who's IP is 69.55.225.225

It's running from /mnt/data1/ns1c-dir

See DNS for more details

Usage and Notes

  • always mounted to backup1 and backup2 via nfs:
backup2:/mnt/data1 on /backup (nfs)
backup2:/mnt/data2 on /backup2 (nfs)
backup2:/mnt/data3 on /backup3 (nfs)
backup2:/mnt/data4 on /backup4 (nfs)
backup1:/data on /backup1 (nfs)

Cronjobs

* * * * * /usr/local/www/mgmt/mrtg/mrtg.sh > /dev/null 2>&1

Gathers up data for our mrtg/load graphs 

*/5 * * * * /usr/local/bin/rsync -a root@nat2:/mnt/data1/mrtg/data/ /usr/local/www/mgmt/mrtg/data/

Gathers up data from i2b servers for our mrtg/load graphs 

40 0 * * * /usr/local/bin/rsync -a root@nat2:"/mnt/data1/mrtg/*.cfg" /usr/local/www/mgmt/mrtg

Gathers up mrtg configuration (port names) from i2b switches for our mrtg/load graphs 

41 0 * * * for f in `grep -l "mnt\/data1" /usr/local/www/mgmt/mrtg/switch-p*.cfg`; do cat $f | sed s#\/mnt\/data1#\/usr\/local\/www\/mgmt# > $f.new; mv $f.new $f; done

Gathers up mrtg configuration (port names) from castle switches for our mrtg/load graphs 

1 0 1 * * cp /usr/local/www/mgmt/html/top20ip /usr/local/www/mgmt/html/top20ip_last
1 0 1 * * cp /usr/local/www/mgmt/html/top20customers /usr/local/www/mgmt/html/top20customers_last
2 * * * * /usr/local/www/cronjobs/top20ip.pl > /dev/null 2>&1
15 * * * * /usr/local/www/cronjobs/top20customer.pl > /dev/null 2>&1
1 0 1 * * rm /usr/local/www/mgmt/html/bandtrack

Archiving and generation of bandwidth statistics presented in mgmt -> Reference -> Bandwidth 

1 0 * * * /usr/local/etc/rsync.backup

Nightly backup script 

0 1 * * * /usr/local/www/mgmt/awstats/wwwroot/cgi-bin/awstats.pl -config=jcpub -update

Public web traffic stats 

15 0 * * * rm /usr/local/www/mgmt/bwgraphs/*.png
16 0 * * * rm /usr/local/www/am/bwgraphs/*

Cleanup for graph-related temp data generated by customers using the bandwidth reports via the AM 

10 0 1 * * /usr/local/www/cronjobs/monthly_bandwidth_report.pl

Monthly bandwidth overage report 

*/3 * * * * /usr/local/www/cronjobs/bbcheck.pl

Updates mgmt with bb monitoring issues 

5 0 * * * /usr/local/www/cronjobs/shutdownreminder.pl

Emails customers reminding them of upcoming shutdown date 

7 0 * * * /usr/local/www/cronjobs/invoice_email.pl

Emails customers who have invoices and are set to auto-email (currently no customer gets these) 

8 */4 * * * /usr/local/www/cronjobs/mysqlrepchk.pl

Checking that we are properly replicating (mysql) traffic data from bwdb to backup1 

16 0 1 * * /usr/local/www/cronjobs/purge_traffic.pl

Removed old traffic data from the traffic database (running on backup1) 

*/5 * * * * chmod 0700 /usr/local/www/ccard_orders/* && mv /usr/local/www/ccard_orders/* /usr/local/www/ccard_orders/done

Secure credit card data: set root-read-only 

25 0 * * * /usr/local/www/cronjobs/biller.pl

Enters service charges in customer billing ledgers 

10 13 * * * /usr/local/www/cronjobs/pfp_batch_gather.pl

Looks for customers with balance due and active credit card on file, prepares a payflow batch 

10 14 * * * /usr/local/www/cronjobs/pfp_batch_process.pl

Tries to collect ccard funds for items in payflow batch - communicates with payflow 

0 7 * * 1 /usr/local/www/cronjobs/email_pmt_reminder.pl

Emails customers in arrears, reminding them to pay 

0 0 1 * * /usr/bin/mail -s 'archive sent mail in pine' support@johncompanies.com < /dev/null

Reminds us to archive sent mail 

0 3 * * * /usr/local/bin/rsync -a isys.e-monitoring.net:/var/mail /backup2/isys; /usr/local/bin/rsync -a isys.e-monitoring.net:/usr/home /backup2/isys

Backup data on isys

Regular maintenance

bwdb

Summary

This machine tracks and stores network traffic (netflow). It is our means to monitor customer bandwidth usage.

  • Location: castle, cab 3-7
  • OS: FreeBSD 4.10 x86
  • Networking: Priv IP: 10.1.4.203 There are 2 onboard nic's, one of which is the "listener"
  • Hardware: Custom 1U. Single power supply.
  • Drives: two 250 GB (2 x 250GB) RAID1 arrays running on a Promise IDS RAID card.

Services Provided

  • netflow
  • mysql
  • bigbrother
  • snmp

firewall (newgateway)

Summary

This machine is the primary (only) firewall for the entire network at castle.

  • Location: castle, cab 3-8
  • OS: FreeBSD 4.11 x86
  • Networking: Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal). It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the right (em1), and the external-facing-network (3750) is on the left (em0).
  • Hardware: 6 SCSI SCA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
  • Drives: 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card.

Services Provided

  • firewall (ipfw)
  • snmp
  • bigbrother

Firewall Rule Configuration

See FreeBSD_Reference#Firewall_Rule_Configuration for more discussion on how to actually manipulate firewall rules.

Disaster Recovery

If there is ever an outage with the firewall, the old firewall "gate" is located just below and is running with the proper network configuration, but with no firewall rules in place (to facilitate good throughput). Have castle move the cable on the left on the current firewall to the left port in the old firewall and the right cable to the right port.

Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs) 

hostname="newgateway.johncompanies.com"
firewall_script="/etc/firewall.sh"
firewall_enable="NO"
sendmail_enable="NONE"
sshd_enable="YES"
inetd_enable="NO"
xntpd_enable="YES"
snmpd_enable="YES"
#snmpd_flags="-as -p /var/run/snmpd.pid"
#ipnat_enable="YES"
#ipnat_rules="/etc/ipnat.rules"
gateway_enable="YES"

defaultrouter="69.55.233.161"

ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0"
ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248"

#
# Original JohnCompanies 69.55.224.0/20
#
ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248"

static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1
2 route13 route14 route15 route16 route17 route18"

route_route1="-net 69.55.224.0 69.55.233.153"
route_route2="-net 69.55.225.0 69.55.233.153"
route_route3="-net 69.55.226.0 69.55.233.153"
route_route4="-net 69.55.227.0 69.55.233.153"
route_route5="-net 69.55.228.0 69.55.233.153"
route_route6="-net 69.55.229.0 69.55.233.153"
route_route7="-net 69.55.230.0 69.55.233.153"
route_route8="-net 69.55.231.0 69.55.233.153"
route_route9="-net 69.55.232.0 69.55.233.153"
route_route10="-net 69.55.233.0 69.55.233.153"
route_route11="-net 69.55.234.0 69.55.233.153"
route_route12="-net 69.55.235.0 69.55.233.153"
route_route13="-net 69.55.236.0 69.55.233.153"
route_route14="-net 69.55.237.0 69.55.233.153"
route_route15="-net 69.55.238.0 69.55.233.153"
route_route16="-net 69.55.239.0 69.55.233.153"
route_route17="-net 10.1.5.0 10.1.4.2"
route_route18="-net 10.1.6.0 10.1.4.2"


#In case of 3750 failure:
#defaultrouter="69.43.128.81"
#ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248"

#bind .1's here:
#ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0"
#ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0"
#ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0"
#ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0"
#ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0"
#ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0"
#ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0"
#ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0"
#ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0"
#ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0"
#ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0"
#ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0"
#ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0"
#ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0"
#ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0"
#ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0"

#bulk:
# reassign 69.55.231.1 to the int iface on the firewall
# set the DG on the firewall to 69.43.138.9
# set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248

Cronjobs

1 0 * * * /usr/local/etc/rsync.backup

Backup to backup1 

0 0 1 * * /sbin/ipfw zero
0 0 1 * * /sbin/ipfw del 3  4 5 17331

Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).

Inside /etc/daily.local you will see a call to /etc/makepiperules.pl This script will create /etc/firewall.sh which contains all the firewall and pipe rules in place at the time the script was run.

DOS attacks

See FreeBSD_Reference#Handling_a_DoS_attack regarding how to handle a DOS attack.

Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will

  1. enter a rule that allows all UDP to go through
  2. send an emergency email to support and indicating an attack is in progress
  3. send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
  4. wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1

This file lives under /usr/home/user/doswatch.pl To run: 

cd /usr/home/user
./doswatch.pl &

To kill; 

fg
^C

It writes its findings to /usr/home/user/doswatch.log

backup1

Summary

This machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server.

  • Location: castle, cab 3-8
  • OS: Ubuntu 8.04.1 server x86
  • Networking: Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
  • Hardware: 6 SATA drive bays (2 columns of 3, drive 0 top left, drive 1 just below) all hot-swap. Single power supply.
  • Drives: 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card

Services provided

  • backup via rsync
  • mysql
  • nfs
  • snmp
  • bigbrother

Usage and Notes

  • all data is stored under /data
  • virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: /backup1), and they all have their ssh keys setup to allow passwordless rsync's
  • each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
  • at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
  • cancelled customer systems are compressed and stored under /data/deprecated
  • archived bwdb2 flow files are stored under /data/bwdb2
  • critical files from backup2 are stored under /data/backup2

Cronjobs

00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null
35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate  /data/backuplog.log

this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup. 

10,25,40,55 * * * * /usr/local/sbin/processsql.pl

this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database. 

0 0 * * * /usr/local/sbin/3wraidchk

checks the health of the RAID array

Regular maintenance

backup2

Summary

This machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. NOTE: power button is broken, so the reset button (paper clip) was rewired to be the power button.

  • Location: castle, cab 3-7
  • OS: FreeBSD 6.1 x86
  • Networking: Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC infrastructure @ i2b)
  • Hardware: 16 IDE drive bays (4 columns of 4, drive 0-0 top left, drive 0-1 just to the right TODO) all hot-swap. Triple power supply.
  • Drives:
    • 3ware 7500-8:
      • 200 GB JBOD (1 x 200G) labeled 0-0
      • 500 GB RAID5 (3 x 250G) 0-1 thru 0-3
      • 700 GB RAID5 (4 x 250G) 0-4 thru 0-7
    • 3ware 7500-8:
      • 700 GB RAID5 (4 x 250G) 1-0 thru 1-3
      • 700 GB RAID5 (4 x 250G) 1-4 thru 1-7

All drives MUST be western digital IDE drives. Other brands will not fit.

Services provided

  • backup via rsync and nfs
  • samba
  • nfs
  • snmp
  • bigbrother

Usage

  • all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: /mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4
  • iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /mnt/data2/iso
  • this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
  • this server serves as an archive for exported db data from bwdb and old flow files.
  • isys backs up here
  • customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
  • in /mnt/data4 there are lots of useful things used for building our vps servers, customer servers, and management scripts:
    • /bin: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
    • /build: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
    • /vzrpms: contains the OS templates for many-to-most of the OS's we offer on vz systems

Cronjobs

  • backs itself up nightly to nfs-mounted backup1 (mountpoint: /backup2)

Regular maintenance

backup3

Summary

This machine is used for archiving data, is a backup server for colo customers, runs a samba server to make available iso's to the IPKVMs, and allows us to connect to the digi serial multiplexer at i2b. Only dedicated customers directly accesses this server to perform their backups.

  • Location: i2b, cab 6
  • OS: Ubuntu 10.04.1 server amd64
  • Networking: Priv IP: 10.1.2.3, Pub IPs: 69.55.229.4 AND 69.55.231.2
  • Hardware: 16 drive SATA bays (4 columns of 4, drive 0 top left, drive 1 just below) all hot-swap. Dual power supply.
  • Drives: 5 TB (6 x 1TB) RAID5 array running on an Areca Technology Corp. ARC-1160 16-Port

Services provided

  • backup via rsync and nfs
  • samba
  • nfs
  • digi realport
  • snmp

Usage

  • all data is stored under /data
  • iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /data/iso
  • this server serves as an archive for exported db data from bwdb and old flow files.
  • inftrastructure machines at i2b back up here
  • customers are nfs-moutned under /data/customers as file-backed loopback devices

management scripts

  • mkbackups

Cronjobs

0 0 * * * /usr/local/sbin/arecaraidchk

RAID checks 

35 4 * * * /usr/local/sbin/snapshot_archive

Rotate daily snapshots for infrastructure machine backups

Regular maintenance

Build

BIOS Config

disable quiet boot

set to last state after power loss

set date/time to GMT

enable serial console output (baud rate 115200)

Install OS

Ubuntu 10.04.1 amd64 (couldn't get 12.04 to load cause the H/W was incompat)
10G / ext3
2G swap
~ /data ext4

Install packages:
openssh
samba

DNS and private IP

echo "nameserver 69.55.225.225" >> /etc/resolv.conf

Add a 2nd IP to eth0 and setup priv net 

vi /etc/network/interfaces

auto eth0
iface eth0 inet static
        address 69.55.229.4
        netmask 255.255.255.0
        network 69.55.229.0
        broadcast 69.55.229.255
        gateway 69.55.229.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 69.55.229.3 66.181.0.2
        dns-search johncompanies.com

auto eth0:1
iface eth0:1 inet static
        address 69.55.231.2
        netmask 255.255.255.0
        network 69.55.231.0
        broadcast 69.55.231.255

auto eth1
iface eth1 inet static
        address 10.1.2.3
        netmask 255.255.255.0
        network 10.1.2.0
        broadcast 10.1.2.255

Install packages

apt-get update
apt-get upgrade
apt-get install gcc
apt-get install libssl-dev
apt-get install libncurses5-dev
apt-get install cu
apt-get install unzip
apt-get install snmp snmpd ntp nfs-kernel-server

tweak grub, enable serial

vi /etc/default/grub
#GRUB_HIDDEN_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="max_loop=64"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
update-grub
echo "start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L ttyS0 38400 vt102" > /etc/init/ttyS0.conf

install realport (digi) driver

give the digi an ip with DgIpServ.exe 

cd /usr/src/
wget ftp://ftp1.digi.com/support/beta/linux/dgrp/dgrp-1.9.tgz
tar xzf dgrp-1.9.tgz 
cd dgrp-1.9/
./configure
make
make install
make postinstall
update-rc.d dgrp_daemon defaults

configure ports: 

dgrp_cfg_node init el 10.1.2.10 16

try connecting with: 

cu -l /dev/ttyel00 -s 38400

shell, ntp, ssh key, hosts

Shell autocompletion search: 

echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
echo "\"\e[6~\": history-search-forward" >> ~/.inputrc

Setup ntp: 

vi /etc/ntp.conf
server 10.1.2.1
server ntp.ubuntu.com

Generate ssh keys: 

cd /root/
ssh-keygen -t dsa

Defaults, no password

Setup hosts: 

echo "69.55.230.10 backup2" >> /etc/hosts
echo "69.55.230.11 backup1" >> /etc/hosts
echo "10.1.2.4 bwdb2" >> /etc/hosts
echo "10.1.2.3 backup3" >> /etc/hosts

Copy keys to servers where we need passwordless login: 

cat .ssh/id_dsa.pub | ssh backup2 'cat - >> /root/.ssh/authorized_keys' 
cat .ssh/id_dsa.pub | ssh backup1 'cat - >> /root/.ssh/authorized_keys'

Setup shell: 

vi /root/.bashrc
(add to bottom)
alias h='history'
alias vi='vim'
alias j='jobs'
export PS1="[\u@\h \w]# "
alias dr='screen -dr'
export EDITOR=vim
export GREP_OPTIONS='--color=auto'
export HISTFILESIZE=1000

alias tip-switch-p20='cu -l ttyel00 -s 9600'
alias tip-switch-p21='cu -l ttyel15 -s 9600'
alias tip-switch-p22='cu -l ttyel14 -s 9600'
alias tip-switch-p23='cu -l ttyel05 -s 9600'
alias tip-switch-p24='cu -l ttyel06 -s 9600'
alias tip-switch-p25='cu -l ttyel09 -s 9600'
alias tip-switch-p26='cu -l ttyel07 -s 9600'
alias tip-switch-p27='cu -l ttyel08 -s 9600'
alias tip-firewall2='cu -l ttyel01 -s 115200'
alias tip-nat2='cu -l /dev/ttyel02 -s 115200'
alias tip-backup3='cu -l ttyel04 -s 38400'
alias tip-bwdb2='cu -l ttyel03 -s 115200'
alias tip-backup4='cu -l ttyel13 -s 115200'
alias tip-jail3='cu -l ttyel11 -s 115200'

Load new shell:
 source /root/.bashrc

Setup snmpd (this is only valid for a server at castle):
echo 'rocommunity  jcread 10.1.4.5
rocommunity  jcread 10.1.4.3
agentaddress 10.1.4.8:161' > /etc/snmp/snmpd.conf

to see which iface it is, on backup2:

snmpwalk -v 1 -c jcread 10.1.4.8 interface

=== nfs ===

Allow mounts from private net:
 echo '/data 10.1.2.0/24(rw,no_root_squash,async,no_subtree_check)' >> /etc/exports

Restart nfsd:
 /etc/init.d/nfs-kernel-server restart

=== bb ===

Add user, group:
 echo "bb:x:1984:1984:Big Brother:/home/bb:/bin/bash" >> /etc/passwd
 echo "bb:x:1984:" >> /etc/group
 pwconv

Create home:
 mkdir /home/bb
 chown bb.bb /home/bb
 cd ~bb

Copy over and install files:
<pre>scp backup2:/mnt/data4/build/bb/bb-linux.tar .
tar xf bb-linux.tar
cd /home/bb/bbc1.9e-btf/etc

Configure main bb server: 

echo "69.55.230.2 mail.johncompanies.com # BBPAGER BBNET BBDISPLAY smtp ssh" > /home/bb/bbc1.9e-btf/etc/bb-hosts
echo "10.1.2.3 backup3.johncompanies.com # ssh" >> /home/bb/bbc1.9e-btf/etc/bb-hosts

Configure low disk alerts: 

echo "/:90:95
/var:90:95
/data:85:99" > /home/bb/bbc1.9e-btf/etc/bb-dftab
vi /home/bb/bbc1.9e-btf/bin/bb-disk.sh

(remove all | SORT xxxx since SORT is broken) 

chmod +r /var/log/messages

./bbchkcfg.sh

(y to questions) 

./bbchkhosts.sh

(ignore ssh errors) 

cd ../..
chown -R bb .
su bb
cd
cd bbc1.9e-btf/src
make; make install
cd ..
./runbb.sh start
more BBOUT

(look for errors) 

exit

vi /etc/rc.local
su - bb -c "cd /home/bb/bbc1.9e-btf; ./runbb.sh start"

(before the exit 0) 

echo 'chmod o+r /var/log/messages' >> /etc/cron.weekly/sysklogd

Add f/w rule: 

ipfw add 00096 allow ip from { 69.55.229.4 or 69.55.229.3 } to 69.55.230.2 1984

vi ~bb/bbc1.9e-btf/etc/bbdef-client.sh
DFWARN=199
DFPANIC=199


raid check

3ware

scp backup1:/usr/local/sbin/tw_cli /usr/local/sbin/tw_cli
scp backup1:/usr/local/sbin/checkraid.sh /usr/local/sbin/checkraid.sh
scp backup1:/usr/local/sbin/3wraidchk /usr/local/sbin/3wraidchk
vi /usr/local/sbin/checkraid.sh
:%s/c0/c2/g

crontab -e
0 0 * * * /usr/local/sbin/3wraidchk

areca

cd /tmp
wget http://www.areca.us/support/s_linux/cli/linuxcli_V1.10.0_120815.zip
unzip linuxcli_V1.10.0_120815.zip
cp linuxcli_V1.10.0_120815/x86_64/cli64 /usr/local/sbin/
chmod 0700 /usr/local/sbin/cli64
cli64 rsf info

scp backup2:/data4/bin/arecaraidchk /usr/local/sbin
scp backup1:/usr/local/sbin/Sendmail.pm /usr/local/sbin

crontab -e
0 0 * * * /usr/local/sbin/arecaraidchk

cat > /root/verify.sh
cli64 vsf info
cli64 rsf info
cli64 disk info
cli64 event info
echo press enter when ready to run verify ; read x

cli64 vsf check vol=1

misc binaries

scp backup1:/usr/local/sbin/snapshot_archive /usr/local/sbin/snapshot_archive
vi /usr/local/sbin/snapshot_archive

(remove entries) 

crontab -e
35 4 * * * /usr/local/sbin/snapshot_archive

scp backup1:/usr/local/sbin/pagedave /usr/local/sbin/pagedave
scp backup1:/usr/local/sbin/taskdone /usr/local/sbin/taskdone

Since installing /bin/mail requires all sorts of packages (lame) we write a simple one here...which can only email johncompanies.com addr's unless you add relaying for this host: 

cat > /bin/mail
#!/usr/bin/perl
use strict;
use warnings;

use lib '/usr/local/sbin';
use Sendmail qw(sendmail);

my $sub = $ARGV[1];
my $to = $ARGV[2];


my %mail = (
   To      => $to,
   From    => $to,
   Subject => $sub,
   Message => '',
   smtp    => 'mail.johncompanies.com'
);
sendmail(%mail) || print "Error: $Sendmail::error";


chmod 0700 /bin/mail

mkbackup

mkdir /data/customers

cat > /usr/local/sbin/mkbackup
#!/bin/sh

if test $1; then
  cid=$1
else
  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
  exit
fi

if test $2; then
  gb=$2
else
  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
  exit
fi

if test $3; then
  ip=$3
else
  echo "ERROR: Usage: mkbackup cid GB ip  Terminating."
  exit
fi


if test -e /data/customers/${cid}-file; then
  echo "ERROR: /data/customers/${cid}-file exists"
  exit
else
  echo "touch /data/customers/${cid}-file"
  touch /data/customers/${cid}-file
  count=`echo $gb|awk '{print $1*1000}'`
  echo "dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count"
  dd if=/dev/zero of=/data/customers/${cid}-file bs=1024K count=$count
  echo "/sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file"
  /sbin/mkfs -t ext3 -F -j -q /data/customers/${cid}-file
fi

if test -e /data/customers/$cid; then
  echo "ERROR: /data/customers/$cid exists"
  exit
else
  echo "mkdir /data/customers/${cid}"
  mkdir /data/customers/${cid}
  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid"
  mount -o loop /data/customers/${cid}-file /data/customers/$cid
  df -h /data/customers/$cid

  echo "fsck -y /data/customers/${cid}-file" >> /etc/nfs_backup_mounts.sh
  echo "mount -o loop /data/customers/${cid}-file /data/customers/$cid" >> /etc/nfs_backup_mounts.sh
  echo "" >> /etc/nfs_backup_mounts.sh

  echo "/data/customers/$cid $ip/32(rw,no_root_squash,async,no_subtree_check)" >> /etc/exports
  /etc/init.d/nfs-kernel-server restart
  tail /var/log/messages
fi
chmod 0700 /usr/local/sbin/mkbackup

vi /etc/rc.local

add: 

/etc/nfs_backup_mounts.sh

samba

apt-get install samba

vi /etc/samba/smb.conf
comment out any mounts, add
[data]
   read only = yes
   locking = no
   path = /data/iso
   guest ok = yes
/etc/init.d/smbd restart

mkdir /data/iso

Bring over some stuff from backup2 

cd /data/iso
scp backup2:/d2/iso/3wfirmware.iso .
scp backup2:/d2/iso/MD5SUMS .
scp backup2:/d2/iso/bootimg.iso .
scp backup2:/d2/iso/systemrescuecd-x86-0.2.19.iso .
scp backup2:/d2/iso/win98bootcd.iso .
scp backup2:/d2/iso/acronis_bootdisk.iso .
scp backup2:/d2/iso/memtest86-3.2.iso .

Moving from one server to another

Here are the steps you would take to move settings and data from one server to a new backup server:

  • rsync over all /data/customers (we do this cause if we didn't use *-file it would copy over the files AND the data in the mountpoint)
rsync -av --progress --ignore-times *-file root@10.1.2.33:/data/customers/

after umounting all the customers, copy over the (empty) directories separately: 

for f in `find .  -type d`; do rsync -av $f root@69.55.229.25:/data/customers; done
  • copy mount script
[root@backup3 /data/customers]# scp /etc/nfs_backup_mounts.sh root@69.55.229.25:/etc/nfs_backup_mounts.sh
  • copy rc.local
[root@backup3 /data/customers]# scp /etc/rc.local root@69.55.229.25:/etc/rc.local
  • copy /etc/exports
[root@backup3 /data/customers]# scp /etc/exports root@69.55.229.25:/etc/exports
  • edit /etc/hostname on both machines (set current to oldbackup3)
  • edit /etc/network/interfaces (swap IPs).
  • stop mounts from mounting on old and new servers so it doesnt start with reboot right away:
chmod 000 /etc/nfs_backup_mounts.sh
  • reboot both servers @ same time
  • check everything out
  • run /etc/nfs_backup_mounts.sh on new server
  • if switch port changed update mrtg to reflect correct port pub nic is on (on p20):
vi /usr/local/www/mgmt/mrtg/mrtg1.cfg

console

Summary

This box's only purpose is to serve as a means to connect to the digi serial multiplexer boxes at castle. Connect to it using the blue (cisco) ribbon cable with the beige RJ-45 to serial connector, 9600 8N1.

  • Location: castle, cab 3-8
  • OS: SunOS 5.8 (solaris)
  • Networking: Priv IP: 10.1.4.4
  • Hardware: Sun Netra


To connect to consoles, ssh in as user 'console' and use the tip command to connect to devices listed in /etc/remote

i.e. 

tip switch-p1
tip jail1

Configuring digi/ports

/etc/remote

This is where the configuration/mapping for ports and custom names which we use along with the tip command to connect to various ports on the digi switches.

We have 2 digi's at castle we connect to: 

#3-7 10.1.4.10
virt15:dv=/dev/dty/CO001s:br#38400:el=^C^S^Q^U^D:ie=%$:oe=^D:
virt13:dv=/dev/dty/CO002s:br#115200:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

and 

#3-6 10.1.4.11
jail4:dv=/dev/dty/CP001s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:
jail16:dv=/dev/dty/CP002s:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:hf:

The only things you need to edit are the first part (i.e. jail4) and the speed (i.e. 9600). You can decipher which port on the digi each line corresponds to by the CP001s or CO001s (port 1 on digi1 and digi2), CP002s or CO002s (port 2 on digi1 and digi2)

drpadmin

The tool you use to configure a device to a digi box is drpadmin: 

bash-2.03$ su
Password:
# drpadmin

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit : s
0       10.1.4.10       32      CO      771     never   1027
1       10.1.4.11       32      CP      771     never   1027
2       65.116.11.2     8       el      771     never   1027

Please select an option (a)dd (d)elete (s)how (r)eset (q)uit :

Use those commands above to modify the devices available.

Switching IP/hostname

Edit: 

/etc/defaultrouter
/etc/hosts
/etc/hostname.hme0
/etc/nodename
Maybe needed to run: # ifconfig hme0 10.1.4.4 up
Retrieved from "https://wiki.jcihosting.com/index.php?title=Infrastructure_Machines&oldid=459"