Infrastructure Machines
firewall (newgateway)
Located at castle, this machine is the primary (only) firewall for the entire network at castle. It has 3 network connections (2 onboard, 1 PCI) connecting to the external, internal and private networks. If you're looking at the back of the server, the internal-network-facing nic is on the TODO, and the external-facing-network (3750) is on the TODO.
The server is running FreeBSD 4.11 x86, has a 36 GB (2 x 36GB) RAID1 array running on an Adaptec 2120S PCI RAID card. Both drives are hotswap. Server has dual-power supplies. Priv IP: 10.1.4.223, Pub IPs: 69.55.233.164 (external), 69.55.233.156 (internal).
Services Provided
- firewall (ipfw)
- snmp
Configuration
The firewall has ipfw enabled in the kernel, not loaded as a module, and further, the kernel configuration lines are:
options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=100 options IPFIREWALL_DEFAULT_TO_ACCEPT
Now, the firewall startup script is found here:
/etc/firewall.sh
It is created periodically based on the current ruleset.
The only thing we do with ipfw on the firewall is block or accept packets and occasionally cap some ips (we do not do any counting, or accounting).
The first rule is to allow traffic pointed at the firewall itself to pass – this is to facilitate access in the event of a DoS attack.
00001 allow ip from any to 69.55.230.1
Rules 2-10 are for bandwidth capping and blocking bad people:
00002 pipe 2 ip from 69.55.224.109 to any xmit em0
00003 pipe 3 ip from { 69.55.227.54 or 69.55.227.55 } to any xmit em0
00004 pipe 4 ip from 69.55.238.194 to any xmit em0
00005 pipe 5 ip from 69.55.238.162 to any xmit em0
00006 deny ip from 69.22.167.138 to any
Rule 100 is for our infrastructure machines:
00100 allow udp from any 53 to 69.55.230.2 00100 allow udp from 69.55.230.2 123 to any 00100 allow udp from 69.55.230.2 to any dst-port 53 00100 allow tcp from any to 69.55.230.2 dst-port 22,25,80,443,110,123,1984,8080 setup 00100 allow icmp from any to 69.55.230.2 icmptypes 0,3,8 keep-state 00100 allow udp from 69.55.230.1 161 to 69.55.230.2 00100 deny ip from any to 69.55.230.2 00100 allow tcp from any to 65.55.238.150 dst-port 25 setup
Rules 101-150 are for jails/virts they disable all traffic from the pub net except from mail, backup, dns, and virtuozzo:
00101 deny ip from any to 69.55.238.120
00102 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.53
00102 deny ip from any to 69.55.228.53
00103 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.64
00103 deny ip from any to 69.55.238.64
00104 allow ip from { 69.55.230.2 or 69.55.230.9 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.92
00104 deny ip from any to 69.55.238.92
00106 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.180
00106 deny ip from any to 69.55.238.180
00107 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.210
00107 deny ip from any to 69.55.238.210
00109 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.237.129
00109 deny ip from any to 69.55.237.129
00110 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.236.128
00110 deny ip from any to 69.55.236.128
00111 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.236.92
00111 deny ip from any to 69.55.236.92
00112 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.235.200
00112 deny ip from any to 69.55.235.200
00113 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.225.2
00113 deny ip from any to 69.55.225.2
00114 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.226.128
00114 deny ip from any to 69.55.226.128
00115 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.224.32
00115 deny ip from any to 69.55.224.32
00116 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.224.110
00116 deny ip from any to 69.55.224.110
00117 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.228.2
00117 deny ip from any to 69.55.228.2
00130 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.2
00130 deny ip from any to 69.55.227.2
00132 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.237.220
00132 deny ip from any to 69.55.237.220
00133 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.236.192
00133 deny ip from any to 69.55.236.192
00134 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.236.64
00134 deny ip from any to 69.55.236.64
00135 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.235.170
00135 deny ip from any to 69.55.235.170
00136 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.234.151
00136 deny ip from any to 69.55.234.151
00137 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.225.77
00137 deny ip from any to 69.55.225.77
00138 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.226.2
00138 deny ip from any to 69.55.226.2
00139 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.226.161
00139 deny ip from any to 69.55.226.161
00140 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.224.150
00140 deny ip from any to 69.55.224.150
00141 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.70
00141 deny ip from any to 69.55.227.70
00141 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.229.2
00142 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.70
00142 deny ip from any to 69.55.227.70
00143 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.230.18
00143 deny ip from any to 69.55.230.18
00144 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.229.100
00144 deny ip from any to 69.55.229.100
In addition to rule 00010 (allow all established) and rule 65500 (allow all) we also have a few more special rules:
00012 deny tcp from any to any tcpflags syn tcpoptions !mss 00012 deny icmp from any to any icmptypes 4,5,9,10,12,13,14,15,16,17,18 00012 deny tcp from any to any tcpflags syn,fin 00012 deny tcp from any to any tcpflags fin,psh,rst,urg 00012 allow icmp from any to any 00013 allow udp from any to 69.55.225.225 dst-port 53 00014 deny tcp from any to any dst-port 135
These are the four DoS attack lines we have in place right at the beginning of the ruleset.
When the machine boots, and is running only three total rules, you then log in and run /etc/firewall.sh - that contains all the additional rules - running the script will put them all in place - then the firewall is fully configured.
Now, by default, we do not put any rules in place at all for a customer - they are left wide open. Most customers do not ever change this. However, if a customer requests a ruleset on our firewall, we implement it in the general form that was described above - allow all ports that need to be open, and deny all others.
The firewall rule numbers are not numbered arbitrarily - they are numbered by customer number. So customer 327 gets 03270 - 03279, and customer 589 gets 05890 - 05899 ... once we get to customer 1000, they will have 10000 - 10009.
This does not mean that every customer can only have 10 rules - as you can see from the four DoS attack rules that are all numbered 00003, you can create multiple rules at the same rule number. I don't advise it though.
Because customer requests are generally "allow these and block everything else" we actually have a script on the firewall to create a typical ruleset. The script is called "rulemaker", and it runs like this:
# rulemaker usage: rulemaker [cust#] IP [port1,port2,...,port10]
So, it has three command line options - the customer number (significant digits only), the IP, and a comma-delimited list of ports (with no spaces).
So, if customer 398 comes to you and says:
"please open up tcp ports for ssh, smtp and http and close all the rest"
then you would run:
rulemaker 398 10.10.10.10 22,25,80
And this is what would happen:
gateway# rulemaker 398 10.10.10.10 22,25,80 /sbin/ipfw add 03981 allow udp from 10.10.10.10 to any 53 /sbin/ipfw add 03982 allow udp from any 53 to 10.10.10.10 /sbin/ipfw add 03983 allow tcp from any to 10.10.10.10 22,25,80 setup /sbin/ipfw add 03989 deny ip from any to 10.10.10.10 or, if they have a dns server: /sbin/ipfw add 03981 allow udp from 10.10.10.10 to any 53 /sbin/ipfw add 03982 allow udp from any 53 to 10.10.10.10 /sbin/ipfw add 03983 allow tcp from any to 10.10.10.10 22,25,53,80 setup /sbin/ipfw add 03984 allow udp from any to 10.10.10.10 53 /sbin/ipfw add 03989 deny ip from any to 10.10.10.10 REMEMBER TO ADD YOUR PASTE TO /usr/local/etc/ipfw.sh gateway#
if they have dns, put a 53 in the command line arg to rulemaker
You are shown a list of rules to paste into place if they don't run a dns server, and one if they do.
Note that rulemaker does not actually put any rules in place at all, it just echos the commands you should run. So, since the customer did not specify port 53, we can assume they do not run a dns server, and we can simply paste this:
/sbin/ipfw add 03981 allow udp from 10.10.10.10 to any 53 /sbin/ipfw add 03982 allow udp from any 53 to 10.10.10.10 /sbin/ipfw add 03983 allow tcp from any to 10.10.10.10 22,25,80 setup /sbin/ipfw add 03989 deny ip from any to 10.10.10.10
into the shell, and hit enter once or twice afterwards. Very simple. We then email the customer and tell them that the lines are in place, and to test them.
customer numbers larger than 999 will work fine with this script because:
ipfw add 010000 (rule)
and
ipfw add 10000 (rule)
translate into the same thing. So adding unnecessary zeroes does not hurt anything. (the rulemaker script outputs 0$1 as the rule number - so it always prepends a zero to make the three-digit customer numbers correct, and that zero prepended to a four digit customer number will not hurt anything - it will just be ignored)
Almost every rule in the firewall is part of a little 4 or 5-line set like rulemaker outputs. Some exceptions are when people want you to open up icmp for them as well (since the above rulemaker output denies it) in which case you would simply paste the rulemaker output, and then afterwards add another rule:
ipfw add 03984 allow icmp from any to 10.10.10.10
Remember, if they run a dns server, they need to have tcp port 53 in their port list and you need to paste the second block that rulemaker outputs.
Some customers, however, do not request a formal ruleset - they simply say to block off port 3306 from the outside (mysql) or they say to block all netbios ports (135,137,139) or something like that. If they do this, do not use rulemaker - simply add a rule just for that:
/sbin/ipfw add 05431 deny tcp from any to 10.10.10.10 3306
or
/sbin/ipfw add 05431 deny tcp from any to 10.10.10.10 135,137,139
On the other hand, a customer may request a normal ruleset, but then request that you only open ssh for a certain IP block or IP. Here is an example of a ruleset that was started with rulemaker, but then additional rules were added:
07471 47802 3991038 allow udp from 69.55.225.125 to any 53
07472 14490 1309166 allow udp from any 53 to 69.55.225.125
07473 85950 4252824 allow tcp from any to 69.55.225.125 22,25,53,80,443,110,143,220 setup
07474 45358 3378454 allow udp from any to 69.55.225.125 53
07475 84 5016 allow tcp from any to 69.55.225.127 22,443
07475 94 5472 allow tcp from any to 69.55.225.128 22,443
07476 38805 3552124 allow icmp from any to 69.55.225.127
07476 38524 3536996 allow icmp from any to 69.55.225.128
07478 6 288 allow tcp from 66.166.221.232/29 to 69.55.225.125 3309
07478 286 13728 allow tcp from 66.166.221.232/29 to 69.55.225.125 3306
07479 109767 6222136 deny ip from any to { 69.55.225.125 or dst-ip 69.55.225.127 or dst-ip 69.55.225.128 }
So ... 69.55.225.125 is the main IP, and what was used in rulemaker, and the main allow line is very familiar:
07473 85950 4252824 allow tcp from any to 69.55.225.125 22,25,53,80,443,110,143,220 setup
but then they wanted allow only 22 and 443 to the other two IP addresses:
07475 84 5016 allow tcp from any to 69.55.225.127 22,443 07475 94 5472 allow tcp from any to 69.55.225.128 22,443
(note they share an ipfw rule number)
then icmp should also be allowed to the other two IPs:
07476 38805 3552124 allow icmp from any to 69.55.225.127 07476 38524 3536996 allow icmp from any to 69.55.225.128
then there are two addresses out in the world that should be totally unfettered in their ability to talk to the main IP:
07478 6 288 allow tcp from 66.166.221.232/29 to 69.55.225.125
or to two ports
07478 286 13728 allow tcp from 66.166.221.232/29 to 69.55.225.125 3306 07478 286 13728 allow tcp from 66.166.221.232/29 to 69.55.225.125 3309
(note, again, sharing ipfw numbers, and also specifying a netblock instead of a single IP: 66.166.221.232/29)
then finally, the last rule that rulemaker outputs was thrown out and this was used instead:
07479 109767 6222136 deny ip from any to { 69.55.225.125 or dst-ip 69.55.225.127 or dst-ip 69.55.225.128 }
Since we are dealing with three IPs total.
Some more example requests:
Replacing a rule (customer wants port 21 access):
gateway# g 69.55.225.3 07161 22462 1795170 allow udp from 69.55.225.3 to any dst-port 53 07162 21220 3283214 allow udp from any 53 to 69.55.225.3 07163 52962 2989600 allow tcp from any to 69.55.225.3 dst-port 22,80,443,25,110,995,143,993,53 setup 07164 20234 1314826 allow udp from any to 69.55.225.3 dst-port 53 07169 30715 2409544 deny ip from any to 69.55.225.3 gateway# gateway# ipfw del 07163 ; ipfw add 07163 allow tcp from any to 69.55.225.3 20,21,22,80,443,25,110,995,143,993,53 setup 07163 allow tcp from any to 69.55.225.3 20,21,22,80,443,25,110,995,143,993,53 setup gateway#
Please block all traffic from this range of IPs: Inet num: 195.238.48.0 - 195.238.63.255
gateway# g 69.55.226.144 08441 356 21668 allow udp from 69.55.226.144 to any dst-port 53 08442 6744 1114132 allow udp from any 53 to 69.55.226.144 08443 7358 411368 allow tcp from any to 69.55.226.144 dst-port 22,25,80,110,443 setup 08449 3135 280030 deny ip from any to 69.55.226.144 gateway# gateway# ipfw add 08440 deny ip from 195.238.48.0/20 to 69.55.226.144
in reply, say “your ruleset is now…”
/etc/firewall.sh is backed up daily locally (/etc/oldrules) and to the backup server
We add rules to block traffic from directly contacting our jails/virts. Each rule is basically the same except for the id (which reflects the machine) and the machine’s IP Here’s some examples:
Jail2:
00102 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 69.55.238.150 } to 69.55.238.2
00102 deny ip from any to 69.55.238.2
Quar1:
00130 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.227.2
00130 deny ip from any to 69.55.227.2
Virt12:
00142 allow ip from { 69.55.230.2 or 69.55.230.10 or 69.55.225.225 or 80.89.140.126 or 12.109.148.175 or 69.64.46.27 or 194.67.59.14 or 69.55.238.150 } to 69.55.229.2
00142 deny ip from any to 69.55.229.2
The IPs listed for access are mail (the new mail), backup2, ns1c, and virtuozzo
To dump/watch traffic:
tcpdump –vvv –n –i em1
Disaster Recovery
Here's what you need to put in /etc/rc.conf to get a firewall going (as far as routes and IPs)
hostname="newgateway.johncompanies.com" firewall_script="/etc/firewall.sh" firewall_enable="NO" sendmail_enable="NONE" sshd_enable="YES" inetd_enable="NO" xntpd_enable="YES" snmpd_enable="YES" #snmpd_flags="-as -p /var/run/snmpd.pid" #ipnat_enable="YES" #ipnat_rules="/etc/ipnat.rules" gateway_enable="YES" defaultrouter="69.55.233.161" ifconfig_xl0="inet 10.1.4.223 netmask 255.255.255.0" ifconfig_em0="inet 69.55.233.164 netmask 255.255.255.248" # # Original JohnCompanies 69.55.224.0/20 # ifconfig_em1="inet 69.55.233.156 netmask 255.255.255.248" static_routes="route1 route2 route3 route4 route5 route6 route7 route8 route9 route10 route11 route1 2 route13 route14 route15 route16 route17 route18" route_route1="-net 69.55.224.0 69.55.233.153" route_route2="-net 69.55.225.0 69.55.233.153" route_route3="-net 69.55.226.0 69.55.233.153" route_route4="-net 69.55.227.0 69.55.233.153" route_route5="-net 69.55.228.0 69.55.233.153" route_route6="-net 69.55.229.0 69.55.233.153" route_route7="-net 69.55.230.0 69.55.233.153" route_route8="-net 69.55.231.0 69.55.233.153" route_route9="-net 69.55.232.0 69.55.233.153" route_route10="-net 69.55.233.0 69.55.233.153" route_route11="-net 69.55.234.0 69.55.233.153" route_route12="-net 69.55.235.0 69.55.233.153" route_route13="-net 69.55.236.0 69.55.233.153" route_route14="-net 69.55.237.0 69.55.233.153" route_route15="-net 69.55.238.0 69.55.233.153" route_route16="-net 69.55.239.0 69.55.233.153" route_route17="-net 10.1.5.0 10.1.4.2" route_route18="-net 10.1.6.0 10.1.4.2" #In case of 3750 failure: #defaultrouter="69.43.128.81" #ifconfig_em0="inet 69.43.129.84 netmask 255.255.255.248" #bind .1's here: #ifconfig_em1="inet 69.55.224.1 netmask 255.255.255.0" #ifconfig_em1_alias0="inet 69.55.225.1 netmask 255.255.255.0" #ifconfig_em1_alias1="inet 69.55.226.1 netmask 255.255.255.0" #ifconfig_em1_alias2="inet 69.55.227.1 netmask 255.255.255.0" #ifconfig_em1_alias3="inet 69.55.228.1 netmask 255.255.255.0" #ifconfig_em1_alias4="inet 69.55.229.1 netmask 255.255.255.0" #ifconfig_em1_alias5="inet 69.55.230.1 netmask 255.255.255.0" #ifconfig_em1_alias6="inet 69.55.231.1 netmask 255.255.255.0" #ifconfig_em1_alias7="inet 69.55.232.1 netmask 255.255.255.0" #ifconfig_em1_alias8="inet 69.55.233.1 netmask 255.255.255.0" #ifconfig_em1_alias9="inet 69.55.234.1 netmask 255.255.255.0" #ifconfig_em1_alias10="inet 69.55.235.1 netmask 255.255.255.0" #ifconfig_em1_alias11="inet 69.55.236.1 netmask 255.255.255.0" #ifconfig_em1_alias12="inet 69.55.237.1 netmask 255.255.255.0" #ifconfig_em1_alias13="inet 69.55.238.1 netmask 255.255.255.0" #ifconfig_em1_alias14="inet 69.55.239.1 netmask 255.255.255.0" #bulk: # reassign 69.55.231.1 to the int iface on the firewall # set the DG on the firewall to 69.43.138.9 # set the ext firewall IP to 69.43.138.12, NM: 255.255.255.248
Cronjobs
1 0 * * * /usr/local/etc/rsync.backup
Backup to backup1
0 0 1 * * /sbin/ipfw zero 0 0 1 * * /sbin/ipfw del 3 4 5 17331
Reset counters and remove pipe rules on the 1st of the month. Pay attention when setting up a rule as 3 4 5 (that's not a temporary traffic cap).
Inside /etc/daily.local you will see a call to /etc/makepiperules.pl This script will create /etc/firewall.sh which contains all the firewall and pipe rules in place at the time the script was run.
Doswatch
Theres a background process (running from user shell) that monitors the firewall for incoming UDP DoS attacks. When it notices packets above a certain level it will
- enter a rule that allows all UDP to go through
- send an emergency email to support and indicating an attack is in progress
- send an email to castle (nocstaff@castleaccess.com and jcsupport@castleaccess.com) telling them to investigate and put up a null if warranted
- wait for a couple minutes to see if the attack subsides- if so it will remove the pass-all UDP rule, if not it will repeat the process from #1
This file lives under /usr/home/user/doswatch.pl To run:
cd /usr/home/user ./doswatch.pl &
To kill;
fg ^C
It writes its findings to /usr/home/user/doswatch.log
backup1
Located at castle, this machine acts as the primary backup location for all VPS-based customers. No customer directly accesses this server to perform their backups. We also store cancelled customers on this server. It is running Ubuntu-Server 8.04 x86, and has a 4.5 TB (6 x 1TB) RAID5 array running on a 3ware 9650SE-8LPML (8-port) card. Its drives are hot-swap. Priv IP: 10.1.4.8, Pub IP: 69.55.230.11 (firewalled from all but JC infrastructure @ i2b)
Services provided
- backup via rsync
- mysql
- nfs
- snmp
Usage and Notes
- all data is stored under /data
- virtually all jc infrastructure, and all VPS machines are setup to mount to backup1 via nfs (mountpoint: /backup1), and they all have their ssh keys setup to allow passwordless rsync's
- each virt or jail backs up each evening to backup1. Each server has it's own directory (named for the server). Under those directories are 7 daily snapshots (0-6)
- at the time of writing, the mysql server running here is replicating from (slave to) the mysql instance on bwdb. Requests for bandwidth data usage for customers (coming from management, account manager, and accounting scripts running on mail) all direct towards the database "traffic" running on this server.
- cancelled customer systems are compressed and stored under /data/deprecated
- archived bwdb2 flow files are stored under /data/bwdb2
- critical files from backup2 are stored under /data/backup2
Cronjobs
00 5 * * * /usr/local/sbin/backupwatch.pl 2>&1 > /dev/null 35 5 * * * /usr/local/sbin/usage_check; /usr/local/sbin/snapshot_archive; /usr/local/sbin/snapshot_rotate /data/backuplog.log
this runs daily the scripts to report on how much disk space each customer system occupies and how long their backups took. Then it rotates backups for each system, removing the oldest backup.
10,25,40,55 * * * * /usr/local/sbin/processsql.pl
this processes prepared sql command files sent from/by bwdb2 (@ i2b) and imports them into the traffic database.
0 0 * * * /usr/local/sbin/3wraidchk
checks the health of the RAID array
Regular maintenance
backup2
Located at castle, this machine is used for archiving data and is a backup server for colo customers. It was the former primary backup location for all VPS-based customers before backup1 was installed. Only dedicated customers directly accesses this server to perform their backups. It is running FreeBSD 6.1 x86, and has the following arrays and controllers:
3ware 7500-8:
- 200 GB JBOD (1 x 200G) labeled 0-0
- 500 GB RAID5 (3 x 250G) 0-1 thru 0-3
- 700 GB RAID5 (4 x 250G) 0-4 thru 0-7
3ware 7500-8:
- 700 GB RAID5 (4 x 250G) 1-0 thru 1-3
- 700 GB RAID5 (4 x 250G) 1-4 thru 1-7
All drives MUST be western digital IDE drives. Other brands will not fit. All are hot-swap. Priv IP: 10.1.4.3, Pub IP: 69.55.230.10 (firewalled from all but JC network at i2b and castle)
Services provided
- backup via rsync and nfs
- samba
- nfs
- snmp
Usage
- all data is stored under 4 mount points, corresponding to the 4 large RAID5 arrays: /mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4
- iso images provided for customers wanting to mount an ISO as a CDROM via the IPKVM are provided via samba on this server. Images live under /mnt/data2/iso
- this used to be our primary backup server so you will see old backups from virt and jails around- missing customer data though, just the machine's data
- this server serves as an archive for exported db data from bwdb and old flow files.
- isys backs up here
- customers are nfs-moutned under /mnt/data3/customers as file-backed md devices
- in /mnt/data4 there are lots of useful things used for building our vps servers, customer servers, and management scripts:
- /bin: the master repository of scripts and custom binaries we use on jails and virts. Each night every virt and jail rsync's what's in here to update the local files. So any global updates to scripts would need to be made here (or will be overwritten with what's in here)
- /build: files we use for setting up big brother, 3ware cli and scripts for colo's, vzcp customized setup files and so on
- /vzrpms: contains the OS templates for many-to-most of the OS's we offer on vz systems
Cronjobs
- backs itself up nightly to nfs-mounted backup1 (mountpoint: /backup2)